Aletia Trepte

🔁 Jobs Recovery & Rebuild System

# Save job settings to S3
          key = f"backup_jobs/job_{job_id}_{today}.json"
          
          s3.put_object(
            Bucket=bucket_name,
            Key=key,
            Body=json_data,
            ContentType="application/json"
          )

🔁 Jobs Recovery & Rebuild System

🧩 Tools: Databricks REST API, Python, AWS S3, Boto3, JSON
🔍 Highlights: Pulled job configs + metadata, handled edge cases, backed up by job/date
⚙️ Recovery: Rebuilds jobs and submits via API with latest valid settings
Impact: Reduced recovery time from hours to minutes
Tech Stack: Python, Databricks, REST API, AWS S3, JSON, SQL, Delta, Sigma

🧹 Back4App Class & Data Automation

import 'dotenv/config';
          await fetch(`https://parseapi.back4app.com/classes/TestClass`, {
            method: 'DELETE',
            headers: {
              'X-Parse-Application-Id': process.env.APP_ID,
              'X-Parse-REST-API-Key': process.env.REST_KEY
            }
          });

🧹 Back4App Class & Data Automation

🛠 Setup Scripts: classCreate.mjs and dataUpload.mjs dynamically generated class schemas and populated them with test data.
🧹 Teardown Script: teardown.mjs removed data and deleted classes to reset environments between test runs.
🔐 Config: .env variables with Node.js for secure and flexible setup.
🧪 TDD: Automated clean test environments for reliable iteration.
🔧 Tech Stack: JavaScript (ES Modules), Node.js, REST APIs, Back4App (Parse), Environment Variables

🚚 FreightFlow DBT + Pub/Sub Pipeline

# Simulate freight event stream
          Listening on 
          projects/dbt-learn-bigquery-444608/subscriptions/freight-sub...
          
          Received message: {
            "shipment_id": "feb156c6-788a-4a95", 
            "status_update": "in transit", 
            "last_known_location": "Phoenix, AZ"
          }

🚚 FreightFlow DBT + Pub/Sub Pipeline

📦 Data Modeling: dbt staging, intermediate, and mart models; distance calculation with geopy.
🔄 Real-Time Messaging: Google Pub/Sub with Python scripts for publishing and subscribing.
🧪 Synthetic Data: Generated realistic tracking data with faker + geopy.
🛠 Tech: dbt, DuckDB, Python, Pub/Sub, geopy, faker

🛠️ Corporate Bond Pipeline Deduplication

// Scala: Watermarking, event-time processing
          val ranked = df.withColumn("row_num", row_number().over(
            Window.partitionBy("security_description", "source_id", "record_id")
              .orderBy(when(col("is_C"), 3)
                      .when(col("is_B"), 2)
                      .when(col("is_A"), 1)
                      .otherwise(0).desc)
          )).filter(col("row_num") === 1)

🛠️ Corporate Bond Pipeline Deduplication

⚙️ Pipeline Overhaul: Improved ingestion, joins, and data matching with Scala + SQL.
📊 Optimization: Applied CTEs, window functions, and watermarking for batch control.
🚀 Outcome: Reduced duplicate entries and mismatches in high-volume streaming data.
🧰 Tech: Scala, SQL, Delta Tables, Databricks, AWS S3, JSON, Streaming APIs