My Latest Learning/Creation. I have created my own AI Agents
rajnishspandey
I am
(Rajnish Pandey)
Experienced with Databricks, DBT, SQL, PLSQL, AWS services, ETL, Python Pyspark. I’ve led several automation projects using Python, improving efficiency by 70%.
#DataEngineer
- Designed and implemented scalable ETL/ELT workflows using Apache Spark, Airflow, Databricks, and dbt to streamline data pipelines and improve processing efficiency.
- Collaborated on data validation and quality checks within dbt models on AWS Databricks to ensure data accuracy, consistency, and reliability.
- Led large-scale data transformations using dbt and PySpark, processing large volumes of data while optimizing performance.
- Built and managed Databricks clusters and jobs to execute complex data processing tasks, ensuring optimized performance and resource utilization across multiple environments.
- Leveraged AWS services (S3, IAM, Lambda, EC2, RDS, DynamoDB, SNS, Glue, Athena, Step Functions) to design and implement cloud-based data architectures that are secure, scalable, and efficient.
- Led and mentored project team, managed task allocation, conducted code reviews, and ensured timely delivery of high-quality deployment solutions.
- Optimized SQL, PL/SQL, and Python code, improving performance, reducing execution time, and ensuring maintainable code for data processing tasks.
- Managed code deployments using Jenkins, GitHub/Bitbucket repositories across multiple environments.
- Automated reporting processes using Python and Pandas, creating a reporting tool that increased operational efficiency by 70% and reduced manual intervention.
- Implemented monitoring and alerting mechanisms for Databricks jobs and pipelines to proactively identify issues, ensuring high availability and minimal downtime.
