Source file: Pasted text(20261009-072406).txt # Complete Technical Interview Question & Answer Bank ## Prepared for Pranay Sarode | ETL / Data Validation / Data Engineer > **How to use:** Every item follows **Q# – question** / **A# – answer**. Work top to bottom. Answers are written to be spoken aloud in 30–90 seconds. Replace all `[brackets]` with your real facts before the interview. Never claim production operation of a design you only built in a lab or as a reference architecture. --- ## SECTION 1 — INTRODUCTION & PROFILE QUESTIONS **Q1 — Tell me about yourself (30-second version).** **A1** — Hi, I'm Pranay Sarode from Pune. I have about five and a half years working with data pipelines and data quality, the last three-plus as a Data Engineer at XXX on financial-services and telecom projects. My stack is SQL, Python, PySpark, Azure Databricks and Data Factory, AWS S3 and SSIS. What I'm best at is proving that data is right: source-to-target reconciliation, schema, null and business-rule checks, and finding the root cause when numbers don't match. **Q2 — Tell me about yourself (90-second version).** **A2** — I'm Pranay Sarode, based in Pune. I have about five and a half years in data pipelines, data quality and analytics, the last three-plus as a Data Engineer at XXX. My main project was XXX portfolio analytics in financial services. I built PySpark, Python and SQL pipelines that ingest investment data from Azure Data Lake and AWS S3, then cleanse, transform and reconcile it. A big part of my work was the validation layer: schema validation, null checks, business-rule checks and source-to-target reconciliation, so NAV, risk and fund-performance numbers could be trusted. I also tuned the SQL Server queries behind those KPIs and supported the move of legacy sources to a central Azure platform on Databricks. Before that, on a telecom churn project, I built PySpark ETL, wrote SQL with CTEs and window functions for churn features, and did ETL testing across the lake and warehouse layers, with SSIS for scheduled batches and CI/CD for dev, test and prod. What I enjoy most is the 'why doesn't it match' problem. I narrow it down hop by hop: source, staging, transformation, target, report. I find the exact records, identify the cause, fix it or raise it with evidence, and add an automated check so it cannot silently come back. That is why this data-validation role fits me. **Q3 — Why a validation / QA-focused role?** **A3** — I have been on the building side, and the part that gave the business confidence was the validation work. I want to specialise in it: design the test strategy, automate it in Python and Spark, and own data quality end to end. Building experience makes me a stronger tester because I know where pipelines break: joins, data types, time zones and incremental loads. **Q4 — Why XXX?** **A4** — I spent more than three years at XXX, so I know the delivery model, Agile ceremonies and working with global stakeholders. This role combines that with the validation focus I want. [Add one real, personal reason.] **Q5 — What have you done since November 2024?** **A5** — Since leaving XXX in Nov 2024, I [state your real reason briefly]. I used the time to deepen the platform side: a hands-on lab on RHEL with Hadoop, Hive Metastore, Iceberg, Kafka, Flink, Airflow, Trino and Prometheus/Grafana, and I designed two AWS reference architectures, a production ETL platform and a global stock-market data platform, including how I would test each layer. [Add certificates or projects only if real.] **Q6 — You are a Data Engineer. Why a testing role?** **A6** — Same as Q3. The validation work was the part that gave the business confidence. I want to specialise in test strategy, automation in Python and Spark, and owning data quality end to end. **Q7 — Do you have 4 years of ETL testing?** **A7** — My title was Data Engineer, but validation and reconciliation were part of my deliverables for about five and a half years: schema validation, null checks, business-rule validation, source-to-target reconciliation, ETL testing across lake and warehouse layers. I can show the queries and the framework I would use here. **Q8 — Have you used Informatica, Autosys, Control-M, Snowflake or BigQuery?** **A8** — I haven't used [tool] in a project. The validation approach is the same: counts, then aggregates, then row-level, then business rules. In [tool] I would do that with [equivalent feature]. I have done the same in [tool you know], and I ramp up quickly. Honest and specific beats bluffing every time. **Q9 — Tell me about a defect you missed or a mistake.** **A9** — Pick a real, small one: what you missed, how it was found, what you changed (a new check, a peer review). Never say you have none. **Q10 — Biggest weakness?** **A10** — I dig deep for root cause, so I now timebox an investigation and escalate with what I know. On the technical side, enterprise schedulers such as Autosys and Control-M and shell scripting are lab-level for me and I am closing that gap. Say only what is true. **Q11 — Tell me about a disagreement with a developer or BA.** **A11** — Evidence-first story: you reproduced it, showed the query and sample keys, referenced the mapping rule, and the BA settled the ambiguity. **Q12 — Your CV says [X]. Show me.** **A12** — Be ready to write code for any line: PySpark in the XXX role, ML models, Docker, Scala and R working knowledge, Tableau. If something is thin, scale the claim to what you can show. **Q13 — How do you keep learning?** **A13** — Your RHEL lab (Hadoop, Hive Metastore, Iceberg, Kafka KRaft, Flink, Airflow, Trino, Prometheus and Grafana), the two reference architectures, and your certificates. **Q14 — Four phrases to repeat.** **A14** — (1) "I never trust a green pipeline, only a reconciled one." (2) "I validate at three levels: structure, volume and content." (3) "I look for the first hop where the numbers diverge." (4) "Every defect I find becomes an automated check." **Q15 — Questions to ask the interviewer.** **A15** — (1) Which source systems and target platforms does this project cover? (2) How is testing split between manual and automated today? (3) What does the mapping or requirements document look like, and who owns it? (4) How are defects triaged with the development team? (5) What would success look like in the first 90 days? --- ## SECTION 2 — ROOT CAUSE ANALYSIS & PRODUCTION INCIDENTS **Q16 — How do you approach a data mismatch? (30 seconds)** **A16** — First I quantify it: which metric, how big, since when. Then I scope it by date, source and key to see if there is a pattern. Then I compare counts and sums hop by hop to find the first layer where they diverge. I test the usual suspects there: join fan-out, filters, data types, time zones, incremental windows. I prove it with a query that returns the exact records, then fix or raise the defect with that evidence, and finally add an automated check so it is caught next time. **Q17 — What is your root cause method?** **A17** — Detect → Scope → Bisect → Hypothesise → Prove → Fix → Prevent. (1) Detect: state the problem in numbers — which table, which metric, how big, since when, which run and environment. (2) Scope: all rows or a pattern? Group the mismatches by date, source system, product, file or partition. (3) Bisect: follow the data hop by hop — source → landing/Bronze → staging/Silver → target/Gold → report. Compare count and sum at each hop. (4) Hypothesise using the common-cause checklist. (5) Prove with a query that isolates the exact records (anti-join, duplicate-by-key, row-level diff). (6) Fix the code or config, or raise a defect with evidence, expected vs actual, and a suggested cause. Retest and run regression. (7) Prevent by turning the finding into an automated check, an alert, a mapping-document update or a runbook step. **Q18 — Target has MORE rows than source.** **A18** — Usual causes: join fan-out (one-to-many join), job run twice (not idempotent), duplicate source files, SCD2 history counted as current. Prove it by counting after each join, GROUP BY key HAVING COUNT(*) > 1 on the joined result, duplicate batch_id/load_ts. Fix: dedupe reference with ROW_NUMBER on latest effective date; join on the full key; MERGE instead of INSERT; uniqueness check. **Q19 — Target has FEWER rows.** **A19** — Usual causes: inner join dropping unmatched keys, filter in mapping, rows rejected on cast/format, incremental watermark gap, failed partition, late data. Prove with left anti-join source to target for missing keys; group missing keys by date/source; check reject and quarantine counts. Fix: left join plus unknown member; fix filter or format; overlap window on watermark; reprocess partition. **Q20 — Counts match, amounts differ.** **A20** — Usual causes: rounding and precision (float vs decimal), FX rate date, sign flip, truncated scale, implicit cast. Prove with row-level diff by key; look at the difference pattern (0.01 = rounding; constant ratio = FX or unit). Fix: DECIMAL with explicit scale; round only at the end; align rate-date rule. **Q21 — Unexpected NULLs in target.** **A21** — Usual causes: join-key mismatch (case, trailing spaces, data type), date parse failure, CASE default, empty string vs NULL (Oracle treats an empty string as NULL). Prove with NULL count per layer; trace one NULL row back to raw; test key with TRIM and UPPER. Fix: standardise keys in Silver; explicit schema and date formats; quarantine unparsable rows. **Q22 — Dates off by a day or by hours.** **A22** — Usual causes: time-zone conversion (UTC, IST, exchange time), DST, timestamp vs date truncation, cut-off time. Prove by comparing raw timestamp with stored value for a few rows; look for a constant 5:30 or 24-hour shift. Fix: store UTC plus local; convert explicitly; write the business-date rule in the mapping document. **Q23 — Garbled or split records.** **A23** — Usual causes: encoding (UTF-8 vs Latin-1, BOM), delimiter inside a field, embedded newline, missing quote character. Prove with column count per line; inspect bad rows; Spark corrupt-record column. Fix: set encoding, quote and escape options; quarantine malformed lines. **Q24 — Job green but output empty or partial.** **A24** — Usual causes: upstream file late or empty, wrong date parameter, swallowed exception, partition not written. Prove with input row count vs output; run parameters; file timestamps; logs. Fix: pre-checks (file present, row count above zero, header/trailer match); fail fast; alert. **Q25 — Job failed.** **A25** — Usual causes: dependency not met, credential or token expiry, schema drift, out of memory from skew, lock or timeout, wrong environment config. Prove by reading the first error in the log, not the last; check upstream status; ask what changed (code, config, source, volume). Fix: fix cause; rerun idempotently; retry with backoff; alert; runbook step. **Q26 — SCD2 dimension wrong.** **A26** — Usual causes: more than one current row, overlapping or gapped dates, hash misses a tracked column, same-day multiple changes, late-arriving change. Prove with SCD2 queries (see SQL section). Fix: fix end-dating logic; include all tracked columns in the hash; order changes by effective timestamp. **Q27 — Duplicate keys in Snowflake, Redshift or BigQuery.** **A27** — Usual causes: primary and unique keys are not enforced on these platforms; retries re-inserted rows. Prove with GROUP BY key HAVING COUNT(*) > 1. Fix: MERGE/upsert; dedupe step; uniqueness test in the pipeline. **Q28 — Report differs from warehouse.** **A28** — Usual causes: filter or slicer context, relationship cardinality or duplicates in a dimension, measure definition, different refresh time, row-level security. Prove by rebuilding the number in SQL with the same filters; compare at the lowest grain, then add filters one by one. Fix: fix model relationship or measure; document the definition; add a report-vs-SQL check. **Q29 — Pipeline suddenly slow.** **A29** — Usual causes: data skew, small files, no partition pruning, non-sargable predicate, stale statistics, more volume. Prove with Spark UI (long-tail task, spill); execution plan; row count per partition. Fix: salting, broadcast, AQE; compaction; sargable filter; index and statistics. **Q30 — Upstream schema change broke the load.** **A30** — Usual causes: new or removed column, type change, column order in CSV. Prove by comparing today's schema with the contract; schema history (Delta, registry). Fix: schema validation gate; contract versioning; schema-evolution rules. **Q31 — What makes a good production incident answer?** **A31** — Symptom, scope, evidence, first divergence, root cause, safe fix, regression proof, prevention and stakeholder communication. **Q32 — Story A: totals did not reconcile after a release (XXX).** **A32** — Situation: After a release, reconciliation showed target market value/NAV higher than source for [N] funds on [date]. Task: Find the cause before the numbers reached business users. Action: Compared count and sum layer by layer. Raw matched, curated did not, and the jump came right after the join to the security reference table. A duplicate check on the reference key showed several effective-dated rows per security, so each position was multiplied. I proved it by tracing one security end to end. Result: Joined to the latest effective row using ROW_NUMBER, reran only the affected dates idempotently, and totals matched within [tolerance]. Added two permanent checks: row count must not change across the join, and the reference key must be unique. Likely probes: Why not DISTINCT? (it hides the cause and can drop legitimate rows). How did you know it was the join? (count right before the join, wrong after). **Q33 — Story B: silent NULLs (telecom churn logs).** **A33** — Situation: A source team changed the date format in usage logs. The null check showed [X]% NULL usage_date, only in new files. Task: Find why and stop bad data reaching the churn features. Action: Spark's to_date returns NULL rather than an error when the pattern does not match (with ANSI mode off, the Spark 3.x default), so the job 'succeeded'. I compared a NULL row with its raw line and confirmed the new format. Result: Explicit schema, accepted both patterns with coalesce(to_date(c, 'dd-MM-yyyy'), to_date(c, 'yyyy-MM-dd')), sent unparsable rows to quarantine with an alert, and made the format check the first gate. NULL rate returned to [near 0]%. Likely probe: How would you catch it earlier? (profile each new file and compare null % per column with yesterday). **Q34 — Story C: slow SQL Server KPI query (XXX).** **A34** — Situation: The NAV/KPI query for [report] took [X] minutes. Task: Speed it up without changing results. Action: Read the actual execution plan: scans and key lookups, a function on the date column (YEAR(trade_date) = 2024) making the filter non-sargable, and an implicit conversion between varchar and nvarchar on the join. Rewrote to a date range, aligned data types, added a covering index with INCLUDE columns, and pre-aggregated into a staging table. Result: [X] → [Y] minutes. I proved results were identical by running EXCEPT in both directions on old vs new output (zero rows), and compared logical reads. Likely probe: Why not just add an index? (check the plan first; an index will not fix a non-sargable predicate and it adds write cost). **Q35 — Story D: slow or out-of-memory Spark job (skew).** **A35** — Situation: A nightly PySpark job took [N] minutes with one task far slower than the rest. Action: Spark UI showed one straggler task with a much larger shuffle read. Key distribution showed [X]% of rows had a NULL or 'UNKNOWN' key. I handled NULL keys separately, broadcast the small dimension, enabled AQE skew-join handling, used salting for the remaining hot key, and compacted small output files. Result: Runtime [X] → [Y] minutes and no memory failures. Likely probe: How does salting work? (add a random suffix 0..N-1 to the hot key on the big side, replicate the small side N times with each suffix, join on key plus suffix). **Q36 — Story E: migration where counts matched but values did not.** **A36** — Situation: In a legacy-to-Azure migration, row counts matched but [N] rows differed in amount or text. Action: A row-hash comparison isolated the keys, then a column-by-column diff showed the cause: DECIMAL(18,4) values read as double lost precision, and text differed by trailing spaces and empty string vs NULL. Result: Explicit decimal schema, trim rule, empty-to-NULL rule from the mapping document; rerun gave zero differences. The hash comparison became a standard migration check. Likely probe: Why hash? (cheap for wide tables; then drill down only on differing keys). **Q37 — Story F: green job, zero rows.** **A37** — Situation: The scheduled job succeeded but Gold had no data for [date]; users noticed first. Action: The input folder was empty because the upstream file arrived late; the job read zero rows, wrote nothing and raised no error. Result: Added pre-checks (file exists, size above zero, row count above zero and equal to the trailer count), fail-fast with an alert, a wait-and-retry until the SLA cut-off, and a freshness check on Gold. Missed loads now alert in minutes. Likely probe: Why not just rerun? (rerun fixes today; the process fix and detection prevent tomorrow). **Q38 — How do you perform root cause analysis generally?** **A38** — 5 Whys, fishbone, hop-by-hop comparison (source → Bronze → Silver → Gold → report). Find first hop where numbers diverge. --- ## SECTION 3 — SQL QUESTIONS ### 3.1 Basic SQL Concepts **Q39 — WHERE vs HAVING?** **A39** — WHERE filters rows before grouping; HAVING filters groups after aggregation. **Q40 — UNION vs UNION ALL?** **A40** — UNION removes duplicates (extra sort); UNION ALL keeps them and is faster. Use UNION ALL for reconciliation counts. **Q41 — DELETE vs TRUNCATE vs DROP?** **A41** — DELETE is row-by-row, logged, can have WHERE; TRUNCATE empties the table fast; DROP removes the table. **Q42 — COUNT(*) vs COUNT(col)?** **A42** — COUNT(col) skips NULLs. NULL = NULL is unknown, not true; use IS NULL. **Q43 — What is a sargable predicate?** **A43** — A filter that can use an index. Avoid functions on the column (YEAR(trade_date) = 2024); use a range instead. **Q44 — CTE vs temp table?** **A44** — A CTE is a readable named subquery, usually not materialised; a temp table is materialised and can be indexed, so it suits reuse on large data. **Q45 — Clustered vs non-clustered index?** **A45** — Clustered defines the physical row order (one per table); non-clustered is a separate structure pointing back to the rows. **Q46 — Slow query checklist?** **A46** — Read the actual execution plan, check scans vs seeks, key lookups, implicit conversions, stale statistics, missing or non-covering indexes, row-by-row logic, partition pruning. **Q47 — Normalisation?** **A47** — 1NF atomic values; 2NF no partial dependency on part of a key; 3NF no transitive dependency. Warehouses denormalise on purpose (star schema) for read speed. **Q48 — EXCEPT vs EXCEPT ALL?** **A48** — EXCEPT removes duplicates (like DISTINCT); EXCEPT ALL preserves duplicates. Use EXCEPT ALL for reconciliation where duplicate counts matter. **Q49 — LEFT ANTI JOIN vs NOT IN vs NOT EXISTS?** **A49** — LEFT ANTI is NULL-safe and fast. NOT IN returns no rows if subquery has NULL. NOT EXISTS is safe but sometimes slower. **Q50 — ROW_NUMBER vs RANK vs DENSE_RANK?** **A50** — For salaries 100,100,90: ROW_NUMBER=1,2,3; RANK=1,1,3; DENSE_RANK=1,1,2. Use DENSE_RANK for "second highest". **Q51 — EXCEPT vs MINUS?** **A51** — Same concept; Oracle uses MINUS, SQL Server/PostgreSQL use EXCEPT. **Q52 — What are the SQL dialect differences to know?** **A52** — Oracle: MINUS, NVL, FETCH FIRST, STANDARD_HASH, empty string = NULL. SQL Server: EXCEPT, ISNULL, TOP, HASHBYTES. PostgreSQL: EXCEPT/EXCEPT ALL, COALESCE, LIMIT, MD5. Snowflake/Redshift/BigQuery: EXCEPT/MINUS, COALESCE, LIMIT, HASH/SHA2; PK/FK/UNIQUE not enforced; QUALIFY supported. MySQL: no FULL OUTER JOIN (use UNION of LEFT + RIGHT). **Q53 — What are the 12 validation queries to write from memory?** **A53** — (1) Row counts by date with FULL OUTER JOIN. (2) Aggregate fingerprint (COUNT, COUNT DISTINCT, SUM, MIN, MAX). (3) Missing and extra rows with EXCEPT/MINUS or left anti-join. (4) Column-level null-safe mismatches. (5) Row-hash comparison for wide tables. (6) Duplicate detection by business key. (7) Keep latest, remove rest (ROW_NUMBER, staging only). (8) NULL/blank/padding checks in one pass. (9) Referential integrity/orphans with NOT EXISTS or LEFT JOIN. (10) Domain, range and format checks (status, qty, price, ISIN). (11) Transformation rules from mapping (amount = qty × price). (12) Incremental load and idempotency check. **Q54 — How do you make checks automation-friendly?** **A54** — Every check writes one row to a `dq_results` table (test_name, src_value, tgt_value, status, run_ts). A dashboard or the pipeline gate reads that table. **Q55 — How do you validate against a consistent snapshot?** **A55** — If source and target are queried while a load is running, you get false mismatches. Use an as-of timestamp, snapshot isolation, or wait for the batch-complete flag. Reading with NOLOCK can show uncommitted data and raise false defects. ### 3.2 SQL Coding Questions **Q56 — Write SQL to find duplicate trade IDs.** **A56** ```sql SELECT trade_id, COUNT(*) AS cnt FROM tgt.trades GROUP BY trade_id HAVING COUNT(*) > 1; ``` **Q57 — Write SQL to find source keys missing from target.** **A57** ```sql SELECT s.trade_id FROM src.trades s LEFT JOIN tgt.trades t ON t.trade_id = s.trade_id WHERE t.trade_id IS NULL; ``` **Q58 — Write SQL to reconcile counts by business date.** **A58** ```sql WITH s AS (SELECT trade_date, COUNT(*) AS cnt FROM src.trades GROUP BY trade_date), t AS (SELECT trade_date, COUNT(*) AS cnt FROM tgt.trades GROUP BY trade_date) SELECT COALESCE(s.trade_date, t.trade_date) AS trade_date, COALESCE(s.cnt, 0) AS src_cnt, COALESCE(t.cnt, 0) AS tgt_cnt, COALESCE(t.cnt, 0) - COALESCE(s.cnt, 0) AS diff FROM s FULL OUTER JOIN t ON s.trade_date = t.trade_date WHERE COALESCE(s.cnt, 0) <> COALESCE(t.cnt, 0); ``` **Q59 — Write SQL for null-safe value mismatch.** **A59** ```sql SELECT s.trade_id, s.amount AS src_amount, t.amount AS tgt_amount FROM src.trades s JOIN tgt.trades t ON t.trade_id = s.trade_id WHERE s.amount IS DISTINCT FROM t.amount; -- SQL Server: use explicit null checks. MySQL: NOT (s.amount <=> t.amount) ``` **Q60 — Write SQL for second-highest distinct salary per department.** **A60** ```sql SELECT * FROM ( SELECT e.*, DENSE_RANK() OVER (PARTITION BY dept_id ORDER BY salary DESC) AS rnk FROM employees e) x WHERE rnk = 2; ``` **Q61 — Write SQL for latest record per business key.** **A61** ```sql WITH ranked AS ( SELECT s.*, ROW_NUMBER() OVER (PARTITION BY business_key ORDER BY updated_at DESC, source_sequence DESC) AS rn FROM staging s) SELECT * FROM ranked WHERE rn = 1; ``` **Q62 — Write SQL for OHLC integrity check.** **A62** ```sql SELECT * FROM market_prices WHERE low_price > high_price OR open_price < low_price OR open_price > high_price OR close_price < low_price OR close_price > high_price OR volume < 0 OR close_price <= 0; ``` **Q63 — Write SQL for amount transformation rule.** **A63** ```sql SELECT trade_id, quantity, price, amount, ROUND(quantity * price, 2) AS expected_amount FROM target_trades WHERE ABS(amount - ROUND(quantity * price, 2)) > 0.01; ``` **Q64 — Write SQL for SCD2: more than one current version.** **A64** ```sql SELECT security_id, COUNT(*) AS current_versions FROM dim_security WHERE is_current = 'Y' GROUP BY security_id HAVING COUNT(*) <> 1; ``` **Q65 — Write SQL for SCD2: overlapping validity periods.** **A65** ```sql SELECT a.security_id, a.security_key AS key_a, b.security_key AS key_b FROM dim_security a JOIN dim_security b ON a.security_id = b.security_id AND a.security_key < b.security_key AND a.eff_start_date <= b.eff_end_date AND b.eff_start_date <= a.eff_end_date; ``` **Q66 — Write SQL for SCD2: gaps in validity.** **A66** ```sql SELECT security_id, eff_end_date, next_start FROM ( SELECT security_id, eff_end_date, LEAD(eff_start_date) OVER (PARTITION BY security_id ORDER BY eff_start_date) AS next_start FROM dim_security) x WHERE next_start IS NOT NULL AND next_start <> eff_end_date + 1; ``` **Q67 — Write SQL for referential integrity/orphans.** **A67** ```sql SELECT f.trade_id, f.account_id FROM fact_trades f LEFT JOIN dim_account a ON a.account_id = f.account_id WHERE f.account_id IS NOT NULL AND a.account_id IS NULL; ``` **Q68 — Write SQL for a 7-row moving average.** **A68** ```sql SELECT trade_date, close_price, AVG(close_price) OVER ( PARTITION BY security_id ORDER BY trade_date ROWS BETWEEN 6 PRECEDING AND CURRENT ROW) AS moving_avg_7_rows FROM daily_prices; ``` **Q69 — Write SQL for login streaks of 3+ consecutive days.** **A69** ```sql SELECT user_id, MIN(login_date) AS streak_start, MAX(login_date) AS streak_end, COUNT(*) AS days FROM ( SELECT user_id, login_date, DATEADD(day, -ROW_NUMBER() OVER (PARTITION BY user_id ORDER BY login_date), login_date) AS grp FROM (SELECT DISTINCT user_id, login_date FROM logins) d) g GROUP BY user_id, grp HAVING COUNT(*) >= 3; ``` **Q70 — Write SQL for employees earning more than their manager.** **A70** ```sql SELECT e.emp_name FROM employees e JOIN employees m ON e.manager_id = m.emp_id WHERE e.salary > m.salary; ``` **Q71 — Write SQL for departments with no employees.** **A71** ```sql SELECT d.dept_name FROM departments d LEFT JOIN employees e ON e.dept_id = d.dept_id WHERE e.emp_id IS NULL; ``` **Q72 — Write SQL for top 3 earners per department.** **A72** ```sql SELECT * FROM ( SELECT e.*, ROW_NUMBER() OVER (PARTITION BY dept_id ORDER BY salary DESC) AS rn FROM employees e) x WHERE rn <= 3; ``` **Q73 — Write SQL for running total and 7-day moving average.** **A73** ```sql SELECT order_date, amount, SUM(amount) OVER (ORDER BY order_date ROWS BETWEEN UNBOUNDED PRECEDING AND CURRENT ROW) AS running_total, AVG(amount) OVER (ORDER BY order_date ROWS BETWEEN 6 PRECEDING AND CURRENT ROW) AS moving_avg_7 FROM daily_sales; ``` **Q74 — Write SQL for month-over-month change.** **A74** ```sql SELECT month_start, revenue, revenue - LAG(revenue) OVER (ORDER BY month_start) AS change_amt, ROUND(100.0 * (revenue - LAG(revenue) OVER (ORDER BY month_start)) / NULLIF(LAG(revenue) OVER (ORDER BY month_start), 0), 2) AS pct_change FROM monthly_revenue; ``` **Q75 — Write SQL for latest price as of a date per security.** **A75** ```sql SELECT security_id, price_date, close_px FROM ( SELECT p.*, ROW_NUMBER() OVER (PARTITION BY security_id ORDER BY price_date DESC) AS rn FROM prices p WHERE price_date <= :as_of) x WHERE rn = 1; ``` **Q76 — Write SQL for daily price moves above 20%.** **A76** ```sql SELECT * FROM ( SELECT security_id, price_date, close_px, LAG(close_px) OVER (PARTITION BY security_id ORDER BY price_date) AS prev_close FROM prices) x WHERE prev_close > 0 AND ABS(close_px - prev_close) / prev_close > 0.20; ``` **Q77 — Write SQL for missing trading days per security.** **A77** ```sql SELECT c.cal_date, s.security_id FROM trading_calendar c CROSS JOIN securities s LEFT JOIN prices p ON p.security_id = s.security_id AND p.price_date = c.cal_date WHERE c.is_trading_day = 1 AND p.security_id IS NULL; ``` **Q78 — Write SQL for share of total per department.** **A78** ```sql SELECT dept_id, SUM(salary) AS dept_salary, ROUND(100.0 * SUM(salary) / SUM(SUM(salary)) OVER (), 2) AS pct_of_total FROM employees GROUP BY dept_id; ``` **Q79 — Write SQL for NULL, blank and padding checks in one pass.** **A79** ```sql SELECT COUNT(*) AS total_rows, SUM(CASE WHEN trade_id IS NULL THEN 1 ELSE 0 END) AS null_trade_id, SUM(CASE WHEN account_id IS NULL OR TRIM(account_id) = '' THEN 1 ELSE 0 END) AS null_or_blank_account, SUM(CASE WHEN amount IS NULL THEN 1 ELSE 0 END) AS null_amount, SUM(CASE WHEN account_id <> TRIM(account_id) THEN 1 ELSE 0 END) AS padded_account FROM tgt.trades; ``` **Q80 — Write SQL for row-hash comparison.** **A80** ```sql WITH s AS ( SELECT trade_id, MD5(COALESCE(CAST(account_id AS VARCHAR(50)), '~') || '|' || COALESCE(CAST(ROUND(amount, 2) AS VARCHAR(50)), '~') || '|' || COALESCE(TRIM(status), '~')) AS row_hash FROM src.trades), t AS ( SELECT trade_id, MD5(COALESCE(CAST(account_id AS VARCHAR(50)), '~') || '|' || COALESCE(CAST(ROUND(amount, 2) AS VARCHAR(50)), '~') || '|' || COALESCE(TRIM(status), '~')) AS row_hash FROM tgt.trades) SELECT s.trade_id FROM s JOIN t ON t.trade_id = s.trade_id WHERE s.row_hash <> t.row_hash; ``` **Q81 — Write SQL for incremental load and idempotency.** **A81** ```sql SELECT (SELECT COUNT(*) FROM src.trades WHERE updated_ts > :last_wm AND updated_ts <= :new_wm) AS expected_rows, (SELECT COUNT(*) FROM tgt.trades WHERE load_batch_id = :batch_id) AS loaded_rows; ``` **Q82 — Write SQL for a watermark batch reconciliation.** **A82** ```sql SELECT (SELECT COUNT(*) FROM source_trades WHERE updated_at > :previous_watermark AND updated_at <= :new_watermark) AS expected_count, (SELECT COUNT(*) FROM target_trades WHERE batch_id = :batch_id) AS loaded_count; ``` **Q83 — How do you write a DQ result to a table?** **A83** ```sql INSERT INTO dq_results (test_name, src_value, tgt_value, status, run_ts) SELECT 'trade_row_count', s.c, t.c, CASE WHEN s.c = t.c THEN 'PASS' ELSE 'FAIL' END, CURRENT_TIMESTAMP FROM (SELECT COUNT(*) AS c FROM src.trades) s CROSS JOIN (SELECT COUNT(*) AS c FROM tgt.trades) t; ``` **Q84 — How do you validate ISIN format?** **A84** ```sql SELECT security_id, isin FROM tgt.securities WHERE isin !~ '^[A-Z]{2}[A-Z0-9]{9}[0-9]$'; -- Oracle/Snowflake: REGEXP_LIKE(isin, '^[A-Z]{2}[A-Z0-9]{9}[0-9]$') ``` **Q85 — How do you find fact rows with no matching dimension row?** **A85** ```sql SELECT f.trade_id FROM fact_trades f LEFT JOIN dim_security d ON d.security_key = f.security_key WHERE d.security_key IS NULL; ``` **Q86 — How do you check late-arriving dimension (Unknown member)?** **A86** ```sql SELECT COUNT(*) FROM fact_trades WHERE security_key = -1; ``` **Q87 — How do you check grain (one row per declared grain)?** **A87** ```sql SELECT account_key, security_key, date_key, COUNT(*) FROM fact_positions GROUP BY account_key, security_key, date_key HAVING COUNT(*) > 1; ``` **Q88 — How do you check SCD2 flag and end date agree?** **A88** ```sql SELECT * FROM dim_security WHERE (is_current = 'Y' AND eff_end_date <> DATE '9999-12-31') OR (is_current = 'N' AND eff_end_date = DATE '9999-12-31'); ``` **Q89 — How do you check SCD2 current row equals source on tracked columns?** **A89** ```sql SELECT s.security_id FROM stg.securities s JOIN dim_security d ON d.security_id = s.security_id AND d.is_current = 'Y' WHERE s.name <> d.name OR s.sector <> d.sector; ``` **Q90 — How do you keep latest and remove rest in SQL Server?** **A90** ```sql WITH d AS ( SELECT ROW_NUMBER() OVER (PARTITION BY trade_id ORDER BY load_ts DESC) AS rn FROM stg.trades) DELETE FROM d WHERE rn > 1; ``` **Q91 — How do you keep latest and remove rest in Oracle?** **A91** ```sql DELETE FROM stg_trades WHERE ROWID IN ( SELECT rid FROM (SELECT ROWID AS rid, ROW_NUMBER() OVER (PARTITION BY trade_id ORDER BY load_ts DESC) AS rn FROM stg_trades) WHERE rn > 1); ``` **Q92 — How do you find domain value violations?** **A92** ```sql SELECT status, COUNT(*) FROM tgt.trades WHERE status NOT IN ('NEW', 'SETTLED', 'CANCELLED') GROUP BY status; ``` **Q93 — How do you find range violations?** **A93** ```sql SELECT trade_id FROM tgt.trades WHERE qty <= 0 OR price < 0 OR trade_date > CURRENT_DATE; ``` **Q94 — How do you write a duplicate check with composite key showing rows?** **A94** ```sql SELECT * FROM ( SELECT t.*, COUNT(*) OVER (PARTITION BY account_id, security_id, trade_date, qty, price) AS c FROM tgt.trades t) x WHERE c > 1; ``` **Q95 — What are the SCD test scenarios to quote?** **A95** — Type 1 (overwrite): value updated, no history row, row count unchanged. Type 2 (history): changed attribute creates a new version with a new surrogate key; old version is end-dated; unchanged rows are untouched; facts keep pointing at the version valid on the transaction date. Type 3 (previous value column): previous-value column populated on change. Also test: brand-new key, two changes on the same day, the same file reprocessed (no duplicate versions), late-arriving change, and a delete or inactive flag. --- ## SECTION 4 — PYTHON QUESTIONS ### 4.1 Python Concepts **Q96 — List vs tuple vs set vs dict?** **A96** — list is ordered and mutable; tuple is ordered and immutable; set is unordered and unique; dict maps keys to values with fast lookup. **Q97 — Generator vs list?** **A97** — A generator produces items lazily, so memory stays flat for large files. A list materialises all values in memory. **Q98 — Shallow vs deep copy?** **A98** — Shallow copies the outer object only; deep copies nested objects too. **Q99 — is vs ==?** **A99** — is checks identity; == checks equality of value. **Q100 — Decorator?** **A100** — A function that wraps another function to add behaviour (retry, logging, timing). **Q101 — Context manager (with)?** **A101** — Guarantees cleanup such as closing files and connections. **Q102 — pandas merge vs join vs concat?** **A102** — merge joins on columns, join on the index, concat stacks frames. **Q103 — apply vs vectorised?** **A103** — Vectorised operations are much faster than row-wise apply. **Q104 — Handling missing data?** **A104** — isna, fillna, dropna; decide with the business rule, never silently. **Q105 — Unit testing?** **A105** — pytest fixtures for set-up, parametrise for many inputs, mock for external systems. **Q106 — Exception handling: catch broadly or specifically?** **A106** — Catch expected exceptions at the boundary where you can recover; broad catches can hide defects. Log context and re-raise or fail the run when the system cannot guarantee correctness. **Q107 — Why use logging instead of print in production?** **A107** — Logging supports severity, timestamps, structured context and central collection. Include run IDs and dataset names, but never log secrets or sensitive payloads. **Q108 — Multiprocessing vs threading vs async I/O?** **A108** — Multiprocessing can use multiple CPU cores for CPU-bound work; threads and async I/O can overlap waiting on network/disk operations. Choose based on workload and library behaviour; do not create unbounded concurrency against a rate-limited API. ### 4.2 Python Coding **Q109 — How do you build a config-driven ETL test automation framework?** **A109** — I use pytest with YAML configuration. The YAML defines the table, source/target queries, business keys, and sum columns. The framework dynamically generates tests for each table. **Q110 — Write a Python function to compare two CSV files and list differences.** **A110** ```python import pandas as pd def compare_csv(path_a, path_b, key): a = pd.read_csv(path_a, dtype=str).fillna('') b = pd.read_csv(path_b, dtype=str).fillna('') m = a.merge(b, on=key, how='outer', suffixes=('_a', '_b'), indicator=True) only_a = m[m['_merge'] == 'left_only'] only_b = m[m['_merge'] == 'right_only'] both = m[m['_merge'] == 'both'] diffs = {} for col in [c for c in a.columns if c not in key]: d = both[both[col + '_a'] != both[col + '_b']] if not d.empty: diffs[col] = d[key + [col + '_a', col + '_b']] return only_a, only_b, diffs ``` **Q111 — Write a retry decorator with exponential backoff.** **A111** ```python import time, functools, random def retry(times=3, delay=1.0, factor=2.0, exceptions=(Exception,)): def deco(fn): @functools.wraps(fn) def wrapper(*args, **kwargs): wait = delay for attempt in range(1, times + 1): try: return fn(*args, **kwargs) except exceptions as e: if attempt == times: raise time.sleep(wait + random.uniform(0, 0.25)) wait *= factor return wrapper return deco ``` **Q112 — How do you read a huge file without running out of memory?** **A112** ```python import pandas as pd for chunk in pd.read_csv('big.csv', chunksize=500000): total += chunk['amount'].sum() ``` **Q113 — Write a function to validate header and trailer of a feed file.** **A113** ```python def validate_feed(path): with open(path, encoding='utf-8') as f: lines = [ln.rstrip() for ln in f if ln.strip()] header, body, trailer = lines[0], lines[1:-1], lines[-1] expected = int(trailer.split('|')[1]) assert len(body) == expected, 'Row count mismatch: file=' + str(len(body)) + ' trailer=' + str(expected) return header, len(body) ``` **Q114 — How do you validate fields with regex?** **A114** ```python import re EMAIL = re.compile(r'^[A-Za-z0-9._%+-]+@[A-Za-z0-9.-]+[.][A-Za-z]{2,}$') ISIN = re.compile(r'^[A-Z]{2}[A-Z0-9]{9}[0-9]$') DATE = re.compile(r'^[0-9]{4}-[0-9]{2}-[0-9]{2}$') bad_isin = [r for r in rows if not ISIN.match(r['isin'])] ``` **Q115 — How do you flatten nested JSON?** **A115** ```python def flatten(d, parent='', sep='.'): out = {} for k, v in d.items(): key = parent + sep + k if parent else k if isinstance(v, dict): out.update(flatten(v, key, sep)) else: out[key] = v return out ``` **Q116 — How do you convert time zones safely?** **A116** ```python from datetime import datetime from zoneinfo import ZoneInfo utc = datetime(2026, 10, 8, 18, 30, tzinfo=ZoneInfo('UTC')) ist = utc.astimezone(ZoneInfo('Asia/Kolkata')) ny = utc.astimezone(ZoneInfo('America/New_York')) ``` **Q117 — Write Python word frequency, duplicates, first non-repeating character.** **A117** ```python from collections import Counter def top_words(text, n=3): return Counter(text.lower().split()).most_common(n) def duplicates(items): return [x for x, n in Counter(items).items() if n > 1] def first_non_repeating(s): c = Counter(s) return next((ch for ch in s if c[ch] == 1), None) ``` **Q118 — Write Python dedupe keeping order and two-sum.** **A118** ```python def dedupe_keep_order(items): return list(dict.fromkeys(items)) def two_sum(nums, target): seen = {} for i, x in enumerate(nums): if target - x in seen: return seen[target - x], i seen[x] = i return None ``` **Q119 — Write Python palindrome, anagram, balanced parentheses.** **A119** ```python from collections import Counter def is_palindrome(s): t = ''.join(ch.lower() for ch in s if ch.isalnum()) return t == t[::-1] def is_anagram(a, b): return Counter(a.replace(' ', '').lower()) == Counter(b.replace(' ', '').lower()) def balanced(s): pairs = {')': '(', ']': '[', '}': '{'} stack = [] for ch in s: if ch in '([{': stack.append(ch) elif ch in pairs: if not stack or stack.pop() != pairs[ch]: return False return not stack ``` **Q120 — Write Python to group totals and log error counts.** **A120** ```python from collections import defaultdict, Counter totals = defaultdict(float) for r in rows: totals[r['account_id']] += float(r['amount']) def error_counts(path): c = Counter() with open(path, encoding='utf-8') as f: for line in f: if ' ERROR ' in line: c[line.split(' ERROR ')[1].split(':')[0].strip()] += 1 return c ``` **Q121 — Write Python pandas one-liners for validation.** **A121** ```python df.isna().sum() df.duplicated(subset=['trade_id']).sum() df.describe(include='all') df.groupby('trade_date').agg(rows=('trade_id', 'count'), total=('amount', 'sum')) a.merge(b, on='k', how='outer', indicator=True) ``` **Q122 — How do you compare two datasets without loading a huge table into pandas?** **A122** — Push counts/aggregates/anti-joins to the database or use distributed PySpark; process files in chunks and compare deterministic keys. **Q123 — How do you make a retry loop production-safe?** **A123** — Retry only transient exceptions, add exponential backoff and jitter, cap attempts/elapsed time, log correlation IDs and ensure idempotency. **Q124 — How do you compare decimal financial values safely?** **A124** — Parse monetary values as Decimal or fixed-scale decimal types rather than binary floating point where exact decimal semantics matter. Define currency and rounding rules before applying tolerances. **Q125 — How do you test a data transformation?** **A125** — Unit-test pure transformation functions with normal, null, duplicate, boundary, malformed and timezone cases; add contract/integration tests for storage and orchestration; use reconciliation tests on representative data. Include repeat-run tests for idempotency. **Q126 — Why should configuration be separate from code?** **A126** — It allows the same tested code to run across environments and datasets with controlled parameters. Validate configuration at startup, version it, and keep secrets in a secret manager rather than plaintext configuration files. ### 4.3 Python Framework Skeleton **Q127 — Explain your pytest framework structure.** **A127** ```text etl_tests/ config/mappings.yaml framework/db.py framework/checks.py tests/test_recon.py requirements.txt Jenkinsfile ``` The YAML lists each table, source and target query, business key, mandatory columns and numeric columns. pytest reads that file and generates the same checks for every table: row count, aggregates, missing and extra keys, column-level differences, duplicates, nulls and referential integrity. Failures print the exact keys, secrets come from environment variables or a vault, results go to an HTML and JUnit report, and Jenkins runs the suite after each load so a failure blocks promotion. For big tables I push the comparison into SQL (counts, aggregates, hashes) or use PySpark instead of loading everything into pandas. **Q128 — Write the mappings.yaml file.** **A128** ```yaml tables: - name: trades source_sql: SELECT trade_id, account_id, amount FROM src.trades target_sql: SELECT trade_id, account_id, amount FROM tgt.trades key: [trade_id] not_null: [trade_id, account_id] sum_columns: [amount] ``` **Q129 — Write framework/db.py.** **A129** ```python import os import pandas as pd from sqlalchemy import create_engine, text def get_engine(prefix): return create_engine(os.environ[prefix + '_DB_URL']) def run_query(engine, sql, params=None): with engine.connect() as conn: return pd.read_sql(text(sql), conn, params=params or {}) ``` **Q130 — Write framework/checks.py.** **A130** ```python import numpy as np import pandas as pd def row_count_diff(src, tgt): return {'src': len(src), 'tgt': len(tgt), 'diff': len(tgt) - len(src)} def missing_and_extra(src, tgt, key): m = src[key].merge(tgt[key], on=key, how='outer', indicator=True) missing = m[m['_merge'] == 'left_only'].drop(columns='_merge') extra = m[m['_merge'] == 'right_only'].drop(columns='_merge') return missing, extra def duplicate_keys(df, key): return df[df.duplicated(subset=key, keep=False)] def null_violations(df, cols): return {c: int(df[c].isna().sum()) for c in cols if df[c].isna().any()} def column_mismatches(src, tgt, key, tol=0.0): m = src.merge(tgt, on=key, suffixes=('_src', '_tgt')) out = {} for col in [c for c in src.columns if c not in key]: a, b = m[col + '_src'], m[col + '_tgt'] if pd.api.types.is_numeric_dtype(a) and pd.api.types.is_numeric_dtype(b): bad = ~np.isclose(a, b, rtol=0, atol=tol, equal_nan=True) else: bad = ~((a == b) | (a.isna() & b.isna())) if bad.any(): out[col] = m.loc[bad, key + [col + '_src', col + '_tgt']] return out ``` **Q131 — Write tests/test_recon.py.** **A131** ```python import pathlib, pytest, yaml from framework import db, checks TABLES = yaml.safe_load(pathlib.Path('config/mappings.yaml').read_text())['tables'] @pytest.fixture(scope='session') def src_engine(): return db.get_engine('SRC') @pytest.fixture(scope='session') def tgt_engine(): return db.get_engine('TGT') @pytest.fixture(params=TABLES, ids=lambda t: t['name']) def data(request, src_engine, tgt_engine): t = request.param return t, db.run_query(src_engine, t['source_sql']), db.run_query(tgt_engine, t['target_sql']) def test_row_count(data): t, src, tgt = data r = checks.row_count_diff(src, tgt) assert r['diff'] == 0, t['name'] + ' ' + str(r) def test_no_missing_or_extra_keys(data): t, src, tgt = data missing, extra = checks.missing_and_extra(src, tgt, t['key']) assert missing.empty and extra.empty, 'missing=' + str(len(missing)) + ' extra=' + str(len(extra)) def test_no_duplicate_keys(data): t, src, tgt = data assert checks.duplicate_keys(tgt, t['key']).empty def test_mandatory_columns_not_null(data): t, src, tgt = data assert checks.null_violations(tgt, t.get('not_null', [])) == {} def test_sums_match(data): t, src, tgt = data for c in t.get('sum_columns', []): assert abs(src[c].sum() - tgt[c].sum()) <= 0.01, c def test_column_values_match(data): t, src, tgt = data assert checks.column_mismatches(src, tgt, t['key'], tol=0.01) == {} ``` **Q132 — Write the Jenkinsfile.** **A132** ```groovy pipeline { agent any stages { stage('Install') { steps { sh 'pip install -r requirements.txt' } } stage('ETL validation') { steps { sh 'pytest --junitxml=reports/junit.xml --html=reports/report.html --self-contained-html' } } } post { always { junit 'reports/junit.xml'; archiveArtifacts artifacts: 'reports/**' } } } ``` ### 4.4 Shell Scripting Essentials **Q133 — What shell commands do you use for ETL validation?** **A133** ```bash set -euo pipefail f=/data/in/trades_20261008.csv [ -s $f ] || { echo File missing or empty; exit 1; } wc -l < $f tail -n 1 $f cut -d',' -f1 $f | sort | uniq -d awk -F',' 'NR>1 {c[$5]++} END {for (k in c) print k, c[k]}' $f awk -F',' 'NR>1 {s+=$4} END {print s}' $f awk -F',' 'NR>1 && length($3)==0 {n++} END {print n+0}' $f diff <(sort a.csv) <(sort b.csv) md5sum a.csv b.csv grep -c ERROR job.log; tail -n 100 job.log for x in /data/in/*.csv; do echo $x $(wc -l < $x); done echo $? # cron entry, 2 AM daily: 0 2 * * * /opt/etl/validate.sh >> /var/log/validate.log 2>&1 ``` **Q134 — Write a tiny validation script with a proper exit code.** **A134** ```bash #!/bin/bash set -euo pipefail src_cnt=$(( $(wc -l < src.csv) - 1 )) tgt_cnt=$(psql -t -c 'select count(*) from tgt.trades' | xargs) if [ $src_cnt -ne $tgt_cnt ]; then echo FAIL source=$src_cnt target=$tgt_cnt exit 1 fi echo PASS rows=$src_cnt ``` Caveat: awk and cut split on every comma, so they break on quoted fields that contain commas. For real CSV use Python's csv module or pandas. --- ## SECTION 5 — PYSPARK QUESTIONS ### 5.1 PySpark Concepts **Q135 — Spark vs MapReduce?** **A135** — Spark works in memory with a DAG and is much faster for iterative work; MapReduce writes to disk between stages. **Q136 — Hive vs Spark SQL?** **A136** — Hive is a SQL layer and metastore on Hadoop (MapReduce or Tez); Spark SQL runs on the Spark engine; the Hive Metastore is often shared as the catalog. **Q137 — Narrow vs wide transformations?** **A137** — Narrow transformations (filter, select, map) need no shuffle; wide ones (groupBy, join, distinct) shuffle data across the network and create stage boundaries. DAG leads to stages, stages to tasks, one task per partition. **Q138 — repartition vs coalesce?** **A138** — repartition does a full shuffle and can increase or rebalance partitions; coalesce only reduces partitions without a full shuffle. spark.sql.shuffle.partitions defaults to 200. **Q139 — cache vs persist?** **A139** — cache keeps a DataFrame in memory for reuse; persist lets you choose the storage level. Unpersist when done. **Q140 — Spark DataFrame vs pandas DataFrame?** **A140** — pandas usually processes data on one machine in memory; Spark DataFrames distribute execution across a cluster with lazy optimisation. Do not call collect() on a large Spark DataFrame because it moves data to the driver. **Q141 — Broadcast join vs shuffle join?** **A141** — Broadcast only when the dimension is safely small enough for executor memory; otherwise use shuffle strategies and inspect the plan/UI. **Q142 — Built-in Spark functions vs Python UDFs?** **A142** — Built-ins are visible to Catalyst optimisation and generally avoid Python serialization overhead; Python UDFs are for logic not expressible with built-ins. Prefer built-ins where possible and benchmark UDFs when needed. **Q143 — subtract vs exceptAll?** **A143** — subtract is EXCEPT DISTINCT (ignores duplicates); exceptAll keeps duplicate counts. Use exceptAll for reconciliation. **Q144 — Why does Spark create small files?** **A144** — Excessive output partitions, tiny input files or over-partitioned data can create many small files. Compact data and tune partitioning based on query patterns. **Q145 — What is AQE?** **A145** — Adaptive Query Execution uses runtime statistics to adjust parts of a query plan, such as shuffle partition coalescing and certain skew optimisations. On by default from Spark 3.2. **Q146 — Why use a deterministic tie-breaker in deduplication?** **A146** — If two records have the same timestamp, ordering only by timestamp may select an arbitrary winner. Add source sequence, update version or another documented tie-breaker and test ties explicitly. **Q147 — What is data skew?** **A147** — A few keys own disproportionate records, so one or a few tasks run much longer or spill/OOM. Confirm in Spark UI and key-frequency profiling; consider AQE skew handling, broadcasting a genuinely small dimension, salting hot keys, and separating NULL/unknown keys. **Q148 — How do you make Spark writes rerunnable?** **A148** — Use deterministic keys, batch/partition scope, deduplication and transactional MERGE/overwrite-by-partition semantics appropriate to the table format; test the same batch twice. **Q149 — What is the difference between a transformation and an action?** **A149** — Transformations build a lazy execution plan; actions trigger execution and return results or write output. **Q150 — What is a shuffle and why is it expensive?** **A150** — A shuffle redistributes data across executors, often involving network I/O, serialization and disk spill; joins and aggregations can trigger it. **Q151 — What is Delta Lake?** **A151** — ACID transactions, schema enforcement and evolution, MERGE, time travel, OPTIMIZE with ZORDER, VACUUM. **Q152 — What is Structured Streaming?** **A152** — Checkpoint location stores progress, watermark bounds late data, and an idempotent sink gives effectively-once results. **Q153 — How do you read a slow Spark job?** **A153** — Spark UI stages tab: one long task means skew; spill to disk means memory pressure; huge shuffle read means a wide transformation to review. Use explain() to read the plan. **Q154 — What should you avoid in Spark?** **A154** — collect or toPandas on big data, and Python UDFs where a built-in function exists. ### 5.2 PySpark Coding **Q155 — Write a PySpark script to read with explicit schema.** **A155** ```python from pyspark.sql import SparkSession, Window, functions as F from pyspark.sql.types import (StructType, StructField, StringType, IntegerType, DateType, DecimalType) spark = SparkSession.builder.appName('etl-validation').getOrCreate() schema = StructType([ StructField('trade_id', StringType(), False), StructField('account_id', StringType(), True), StructField('trade_date', DateType(), True), StructField('qty', IntegerType(), True), StructField('price', DecimalType(18, 4), True), StructField('amount', DecimalType(18, 2), True), StructField('_corrupt_record', StringType(), True), ]) src = (spark.read.schema(schema) .option('header', True) .option('mode', 'PERMISSIVE') .option('columnNameOfCorruptRecord', '_corrupt_record') .option('dateFormat', 'yyyy-MM-dd') .csv('abfss://raw@.dfs.core.windows.net/trades/2026-10-07/') .cache()) bad_rows = src.where(F.col('_corrupt_record').isNotNull()) ``` **Q156 — Write PySpark count and aggregate fingerprint.** **A156** ```python def fingerprint(df): return df.agg(F.count('*').alias('rows'), F.countDistinct('trade_id').alias('distinct_ids'), F.sum('amount').alias('total_amount')).first().asDict() ``` **Q157 — Write PySpark missing and extra rows (multiset aware).** **A157** ```python missing = src.select(cols).exceptAll(tgt.select(cols)) extra = tgt.select(cols).exceptAll(src.select(cols)) ``` **Q158 — Write PySpark key-level anti join.** **A158** ```python missing_keys = src.join(tgt, 'trade_id', 'left_anti') ``` **Q159 — Write PySpark column mismatches with null-safe equality.** **A159** ```python j = src.alias('s').join(tgt.alias('t'), 'trade_id') amount_diff = j.where(~F.col('s.amount').eqNullSafe(F.col('t.amount'))) \ .select('trade_id', F.col('s.amount').alias('src_amt'), F.col('t.amount').alias('tgt_amt')) ``` **Q160 — Write PySpark row hash.** **A160** ```python def with_hash(df, cols): parts = [F.coalesce(F.col(c).cast('string'), F.lit('~')) for c in cols] return df.withColumn('row_hash', F.sha2(F.concat_ws('|', *parts), 256)) ``` **Q161 — Write PySpark duplicates check.** **A161** ```python dups = tgt.groupBy('trade_id').count().where('count > 1') ``` **Q162 — Write PySpark NULL profile in one scan.** **A162** ```python nulls = tgt.select([F.sum(F.col(c).isNull().cast('int')).alias(c) for c in tgt.columns]) ``` **Q163 — Write PySpark referential integrity check.** **A163** ```python orphans = tgt.join(spark.table('gold.accounts'), 'account_id', 'left_anti') ``` **Q164 — Write PySpark SCD2 checks.** **A164** ```python w = Window.partitionBy('security_id').orderBy('eff_start_date') dim = spark.table('gold.dim_security').withColumn('next_start', F.lead('eff_start_date').over(w)) gaps_or_overlaps = dim.where(F.col('next_start').isNotNull() & (F.col('next_start') != F.date_add('eff_end_date', 1))) multi_current = (spark.table('gold.dim_security').where(F.col('is_current') == 'Y') .groupBy('security_id').count().where('count <> 1')) ``` **Q165 — Write PySpark profiling in one scan.** **A165** ```python def profile(df): numeric_or_date = {'tinyint', 'smallint', 'int', 'bigint', 'float', 'double', 'decimal', 'date', 'timestamp'} exprs = [F.count('*').alias('row_count')] for c, t in df.dtypes: exprs += [F.sum(F.col(c).isNull().cast('int')).alias(c + '__nulls'), F.approx_count_distinct(c).alias(c + '__distinct')] if t.split('(')[0] in numeric_or_date: exprs += [F.min(c).alias(c + '__min'), F.max(c).alias(c + '__max')] return df.agg(*exprs).first().asDict() ``` **Q166 — Write PySpark deduplicate keeping latest.** **A166** ```python from pyspark.sql import Window, functions as F w = Window.partitionBy('trade_id').orderBy(F.col('load_ts').desc(), F.col('source_sequence').desc()) latest_df = df.withColumn('rn', F.row_number().over(w)).where('rn = 1').drop('rn') ``` **Q167 — Write PySpark salting for skewed join.** **A167** ```python N = 8 big_s = big.withColumn('salt', (F.rand() * N).cast('int')) small_s = small.withColumn('salt', F.explode(F.array([F.lit(i) for i in range(N)]))) joined = big_s.join(small_s, ['key', 'salt']).drop('salt') ``` **Q168 — Write PySpark split good and bad records (quarantine).** **A168** ```python valid = F.col('trade_id').isNotNull() & (F.col('amount') > 0) & F.col('trade_date').isNotNull() ok = F.coalesce(valid, F.lit(False)) good = df.where(ok) bad = df.where(~ok).withColumn('reject_reason', F.lit('failed_basic_rules')) bad.write.mode('append').parquet('abfss://quarantine@.dfs.core.windows.net/trades/') ``` **Q169 — Write PySpark Delta Lake for validation and idempotent loads.** **A169** ```python spark.sql('DESCRIBE HISTORY gold.trades').show() now_df = spark.table('gold.trades') prev_df = spark.sql('SELECT * FROM gold.trades VERSION AS OF 41') changed = now_df.exceptAll(prev_df) ``` ```sql MERGE INTO gold.trades t USING stg.trades s ON t.trade_id = s.trade_id WHEN MATCHED THEN UPDATE SET * WHEN NOT MATCHED THEN INSERT *; ``` **Q170 — Write PySpark word count.** **A170** ```python words = (spark.read.text('input.txt') .select(F.explode(F.split(F.lower(F.col('value')), ' ')).alias('word')) .where(F.col('word') != '') .groupBy('word').count().orderBy(F.desc('count'))) ``` **Q171 — Write PySpark top 3 salaries per department.** **A171** ```python w = Window.partitionBy('dept_id').orderBy(F.col('salary').desc()) top3 = emp.withColumn('rnk', F.dense_rank().over(w)).where('rnk <= 3') ``` **Q172 — Write PySpark running total and previous value.** **A172** ```python w_run = (Window.partitionBy('account_id').orderBy('trade_date') .rowsBetween(Window.unboundedPreceding, Window.currentRow)) w_ord = Window.partitionBy('account_id').orderBy('trade_date') df = df.withColumn('running_amt', F.sum('amount').over(w_run)).withColumn('prev_amt', F.lag('amount').over(w_ord)) ``` **Q173 — Write PySpark join types.** **A173** ```python no_orders = customers.join(orders, 'cust_id', 'left_anti') big_join = big.join(F.broadcast(small), 'key') ``` Types: inner, left, right, full, cross, left_semi (left rows that have a match, left columns only), left_anti (left rows with no match). **Q174 — Write PySpark nulls, conditional columns, cleaning.** **A174** ```python df = (df.fillna({'ccy': 'USD'}) .withColumn('amount_band', F.when(F.col('amount') >= 100000, 'HIGH') .when(F.col('amount') >= 1000, 'MEDIUM').otherwise('LOW')) .withColumn('account_id', F.trim(F.upper(F.col('account_id'))))) ``` **Q175 — Write PySpark dates and time zones.** **A175** ```python df = (df.withColumn('d', F.to_date('date_str', 'dd-MM-yyyy')) .withColumn('ist_ts', F.from_utc_timestamp('utc_ts', 'Asia/Kolkata')) .withColumn('age_days', F.datediff(F.current_date(), F.col('d')))) ``` **Q176 — Write PySpark flatten nested JSON and explode an array.** **A176** ```python raw = spark.read.json('events.json') flat = (raw.select('id', 'user.name', 'user.address.city', F.explode_outer('items').alias('item')) .select('id', 'name', 'city', 'item.sku', 'item.qty')) ``` **Q177 — Write PySpark pivot.** **A177** ```python pv = sales.groupBy('region').pivot('quarter', ['Q1', 'Q2', 'Q3', 'Q4']).sum('amount') ``` **Q178 — Write PySpark schema drift detection.** **A178** ```python def schema_diff(a, b): sa = {f.name: f.dataType.simpleString() for f in a.schema.fields} sb = {f.name: f.dataType.simpleString() for f in b.schema.fields} return {'only_in_a': sorted(set(sa) - set(sb)), 'only_in_b': sorted(set(sb) - set(sa)), 'type_changed': {c: (sa[c], sb[c]) for c in sa if c in sb and sa[c] != sb[c]}} ``` **Q179 — Write PySpark reconciliation by date.** **A179** ```python src_counts = source_df.groupBy("business_date").agg( F.count("*").alias("source_count"), F.sum("amount").alias("source_amount")) tgt_counts = target_df.groupBy("business_date").agg( F.count("*").alias("target_count"), F.sum("amount").alias("target_amount")) recon = (src_counts.join(tgt_counts, "business_date", "full") .fillna(0, subset=["source_count", "target_count"]) .withColumn("count_diff", F.col("target_count") - F.col("source_count")) .withColumn("amount_diff", F.col("target_amount") - F.col("source_amount"))) failures = recon.filter((F.col("count_diff") != 0) | (F.abs(F.coalesce(F.col("amount_diff"), F.lit(0))) > F.lit(0.01))) ``` **Q180 — Write PySpark transformation rule enforcement.** **A180** ```python checked = df.withColumn("expected_amount", F.round(F.col("quantity") * F.col("price"), 2)) violations = checked.filter(F.abs(F.col("amount") - F.col("expected_amount")) > F.lit(0.01)) ``` **Q181 — Write PySpark explain slow job.** **A181** — (1) Inspect Spark UI: longest stage, shuffle read/write, spill, failed/retried tasks and executor loss. (2) Compare key-frequency distributions to detect skew. (3) Inspect df.explain("formatted") for scans, joins and exchanges. (4) Filter/project early; verify partition pruning and file sizes. (5) Broadcast only a dimension proven small enough; consider AQE/skew handling. (6) Re-run reconciliation and performance tests after changes. ### 5.3 PySpark Framework Blueprint **Q182 — Describe your Spark-based DQ framework.** **A182** — (1) Config layer: one YAML/JSON per dataset with source, target, keys, expected schema, rules, tolerances and severity. Each row of the mapping document becomes a rule. (2) Rule library: small, single-purpose functions: schema, count, sum, row-level diff, unique, not-null, domain, range, referential integrity, freshness, profile drift. (3) Runner: executes the rules, catches exceptions per rule (one broken rule must not kill the run) and records metrics plus a sample of failing keys. (4) Results store: a Delta or Parquet table with run_id, dataset, rule, severity, status, source value, target value, failed rows, detail and timestamp. (5) Gate: a CRITICAL failure fails the job so ADF, Databricks Workflows or Airflow stops publication; WARN only alerts. (6) Reporting and alerts: summary report, Power BI view over the results table, email or Teams on failure. (7) CI/CD: run against small fixture data in Jenkins on every code change; run on real data after every load. **Q183 — Write the engine skeleton.** **A183** ```python import uuid from types import SimpleNamespace from datetime import datetime, timezone from pyspark.sql import Row, functions as F RUN_ID = str(uuid.uuid4()) def result(rule, severity, ok, src_v='', tgt_v='', failed=0, detail=''): return Row(run_id=RUN_ID, rule=rule, severity=severity, status='PASS' if ok else 'FAIL', src_value=str(src_v), tgt_value=str(tgt_v), failed_rows=int(failed), detail=str(detail)[:500], run_ts=datetime.now(timezone.utc).isoformat()) def r_schema(ctx, expected): actual = {f.name: f.dataType.simpleString() for f in ctx.tgt.schema.fields} problems = {c: (t, actual.get(c)) for c, t in expected.items() if actual.get(c) != t} return result('schema', ctx.sev, not problems, detail=problems) def r_count(ctx, tol=0): s, t = ctx.src.count(), ctx.tgt.count() return result('recon_count', ctx.sev, abs(s - t) <= tol, s, t, abs(s - t)) def r_sum(ctx, col, tol=0.01): s = ctx.src.agg(F.sum(col)).first()[0] or 0 t = ctx.tgt.agg(F.sum(col)).first()[0] or 0 return result('recon_sum:' + col, ctx.sev, abs(s - t) <= tol, s, t) def r_unique(ctx, key): n = ctx.tgt.groupBy(*key).count().where('count > 1').count() return result('unique:' + ','.join(key), ctx.sev, n == 0, failed=n) def r_not_null(ctx, cols): row = ctx.tgt.select([F.sum(F.col(c).isNull().cast('int')).alias(c) for c in cols]).first().asDict() bad = {c: n for c, n in row.items() if n} return result('not_null', ctx.sev, not bad, failed=sum(bad.values()), detail=bad) def r_orphans(ctx, ref_table, key): n = ctx.tgt.join(ctx.spark.table(ref_table), key, 'left_anti').count() return result('ref_integrity:' + ','.join(key), ctx.sev, n == 0, failed=n) def r_diff(ctx, cols): miss = ctx.src.select(cols).exceptAll(ctx.tgt.select(cols)).count() extra = ctx.tgt.select(cols).exceptAll(ctx.src.select(cols)).count() return result('row_level_diff', ctx.sev, miss == 0 and extra == 0, failed=miss + extra, detail='missing=' + str(miss) + ' extra=' + str(extra)) RULES = {'schema': r_schema, 'count': r_count, 'sum': r_sum, 'unique': r_unique, 'not_null': r_not_null, 'orphans': r_orphans, 'diff': r_diff} def run_suite(spark, cfg): ctx = SimpleNamespace(spark=spark, src=spark.table(cfg['source']), tgt=spark.table(cfg['target']), sev='CRITICAL') out = [] for rule in cfg['rules']: ctx.sev = rule.get('severity', 'CRITICAL') try: out.append(RULES[rule['type']](ctx, **rule.get('args', {}))) except Exception as e: out.append(result(rule['type'], ctx.sev, False, detail='ERROR: ' + str(e))) df = spark.createDataFrame(out) df.write.mode('append').saveAsTable('dq.validation_results') critical = df.where((F.col('status') == 'FAIL') & (F.col('severity') == 'CRITICAL')).count() if critical: raise RuntimeError(str(critical) + ' critical checks failed') return df ``` **Q184 — Write the matching YAML config.** **A184** ```yaml source: stg.trades target: gold.trades rules: - {type: schema, args: {expected: {trade_id: string, amount: 'decimal(18,2)'}}} - {type: count, args: {tol: 0}} - {type: sum, args: {col: amount, tol: 0.01}} - {type: unique, args: {key: [trade_id]}} - {type: not_null, args: {cols: [trade_id, account_id]}} - {type: orphans, args: {ref_table: gold.accounts, key: [account_id]}} - {type: diff, args: {cols: [trade_id, account_id, amount]}, severity: WARN} ``` **Q185 — How do you handle a billion rows?** **A185** — Aggregates and hashes in one pass, partition-wise comparison (by date), and row-level diff only on partitions where aggregates differ. **Q186 — How do you avoid false failures?** **A186** — Tolerances for decimals, consistent rounding, a consistent snapshot of source and target (same cut-off), and normalised types before comparing. **Q187 — How is the framework tested itself?** **A187** — Fixture datasets with planted defects (a duplicate, a NULL, a changed amount, a missing row) must make the matching rule FAIL. **Q188 — Where does it run?** **A188** — A Databricks Workflow or ADF task after the load, and in Jenkins against fixtures on each code change. --- ## SECTION 6 — AWS ARCHITECTURE QUESTIONS ### 6.1 Production ETL Platform on AWS (Image 1) **Q189 — Walk me through the Production ETL architecture.** **A189** — It is a Bronze, Silver, Gold design on S3 for daily enterprise operations. Sources are APIs, SFTP files, RDBMS or CDC, and SaaS events. Ingestion uses EventBridge for schedules and events, Lambda for validation and routing, and Glue connectors for batch extraction, with MSK or Kinesis only if real-time is needed. Raw data lands in Bronze: immutable, append-only, source-aligned, partitioned by date and source, KMS-encrypted, with lifecycle rules. Invalid or corrupt input goes to a Quarantine and DLQ bucket with an SNS alert. Step Functions orchestrates trigger, dependency check, schema validation, incremental bookmark check, parallel Glue jobs, the data quality gate and publish, with retries, timeouts and error handling; MWAA only if the organisation already standardises on Airflow. Silver standardises, deduplicates, joins and enriches, normalises time zone and currency and writes partitioned Parquet. A Great Expectations gate checks schema, nulls, duplicates, outliers and business rules; a critical failure goes to quarantine and stops publication. Gold holds curated business-ready data in Parquet or Iceberg, served through Athena, QuickSight and downstream applications, and Redshift only if required. Operations rest on idempotency, retries with backoff, checkpoints, backfill, replay from immutable Bronze, reconciliation, SNS alerts and audit. Security and governance use IAM, KMS, Secrets Manager, Lake Formation, the Glue Catalog, lineage and CloudWatch, with CI/CD and infrastructure as code. **Q190 — What do you validate in each layer?** **A190** — | Layer | Checks | | --- | --- | | Ingestion | Schedule fires on time; file name, size and checksum; row count vs control file or trailer; metadata added; correct Bronze partition; malformed file goes to quarantine with an alert | | Bronze | Immutable and append-only; content equals source (counts, hash); partitions correct; KMS encryption and lifecycle rules set | | Orchestration | Step order and dependencies; retry, backoff and timeout behaviour; failure path alerts; incremental bookmark has no gaps or overlaps; rerun is idempotent | | Silver | Standardisation rules; key unique after dedupe; joins do not change row counts unexpectedly; time-zone and currency conversions recomputed on samples; late data handled; schema and partitions correct | | DQ gate | Inject bad data and confirm the gate fails, data goes to quarantine and nothing reaches Gold; metrics and reports produced | | Gold | Aggregates reconcile to Silver; KPIs match business definitions; same number in Athena, Redshift and QuickSight; access limited by Lake Formation | | Operations | Run twice gives the same result; backfill of a date range; replay from Bronze; reconciliation job; SNS alert delivered; audit entries exist | | Security and governance | Least-privilege roles; KMS keys and rotation; secret rotation; column-level access; lineage visible in the catalog | **Q191 — What are the design choices and how do you defend them?** **A191** — Immutable Bronze: you can replay and fix logic without re-pulling the source, and it is an audit trail. Quarantine and DLQ: bad records are kept for investigation instead of being lost or poisoning downstream tables. Quality gate before Gold: consumers only ever see validated data. Step Functions first, MWAA only when Airflow is already the standard: less to operate. Auto-scaling and serverless only when volume needs it: cost control. Choose serving targets by workload, not because every engine exists: Athena for ad hoc, QuickSight for BI, Redshift only if a warehouse is needed. Phased delivery: Phase 1 core ETL; Phase 2 reliability; Phase 3 governance; Phase 4 only if required. Stage testing the same way. **Q192 — Why EventBridge and Step Functions?** **A192** — EventBridge schedules or routes events. Step Functions manages multi-step state, dependencies, retries, branches and failure paths. Glue performs ETL; Lambda is best for lightweight event handling, validation or routing, not large distributed transforms. **Q193 — Glue bookmarks vs a custom watermark?** **A193** — Bookmarks track processed data for supported Glue sources/jobs; a custom watermark can encode business-specific CDC semantics. Either way, test late-arriving data, job reruns and checkpoint commit timing. Do not assume a bookmark alone provides end-to-end exactly-once behavior. **Q194 — What is the purpose of KMS and Secrets Manager?** **A194** — KMS manages encryption keys used to protect data; Secrets Manager stores/rotates credentials. IAM policies and key policies both affect access. A key can be correctly configured while a principal still lacks decrypt permission. **Q195 — What would you monitor?** **A195** — Job success/failure and duration; source freshness; rows in/out/rejected; DQ failures; Glue/EMR capacity; MSK consumer lag; S3 partition arrival; serving replica lag; cache age; retries; DLQ depth; report delivery; replication lag; and cost/budget thresholds. **Q196 — What belongs in a DLQ record?** **A196** — Source identifier, event/file/batch ID, timestamp, schema version, error class/message, attempt count, correlation/run ID and a safe payload reference. Avoid storing secrets or unrestricted sensitive data. **Q197 — Why immutable Bronze?** **A197** — It creates a trustworthy audit trail and supports reproducible backfills after code or mapping fixes. Use append-only object paths and access controls; Object Lock is for explicit retention/compliance requirements and must be configured carefully. ### 6.2 Global Investment Banking Stock Market Data Platform (Image 2) **Q198 — Walk me through the Global Stock Market architecture.** **A198** — This is a production-grade market-data platform for real-time plus daily batch. Sources are global exchanges, market APIs, WebSocket tick feeds, corporate actions, reference data such as FX rates and holidays, and news and sentiment. Ingestion has three paths: persistent WebSocket consumers on ECS Fargate with auto-scaling, reconnect with backoff and jitter and schema validation; event-driven Lambda for REST polling, using a central rate-limit manager in DynamoDB so vendor quotas are shared and it fails closed when throttled; and S3-event Lambda or Glue for corporate-action files. Everything publishes to Amazon MSK (multi-AZ, IAM authentication, encryption), with Avro or Protobuf schemas in the Glue Schema Registry and a dead-letter queue that retries up to three times and then quarantines to S3. EMR Serverless runs Spark Structured Streaming and batch ETL: validation, cleansing, enrichment, deduplication by event_id, and writes Bronze, Silver and Gold to S3, with Great Expectations rules for OHLCV, gaps and outliers and a separate backfill path for corrections. S3 is versioned and KMS-encrypted; Bronze uses Object Lock for audit; Silver applies corporate actions; Gold holds daily OHLCV, adjusted prices and aggregates in Delta or Iceberg. Serving is Aurora PostgreSQL for curated time series, ElastiCache Redis for low latency and optional Redshift, with re-sync and cache invalidation after any correction. QuickSight, Athena, SageMaker and notebooks consume it, and SES sends the automated daily reports with an idempotency check. MWAA runs the daily DAG. CI/CD has a schema gate, build and test, Terraform with security scanning, a manual production approval and rollback. It is secured with IAM, KMS and Lake Formation across multiple accounts, and runs multi-region with MSK replication and stated RPO, RTO and freshness targets. **Q199 — What are the targets from the diagram?** **A199** — Streaming RPO under 5 minutes; platform RTO under 1 hour; daily report RTO under 30 minutes; Redshift staleness under 1 hour; Gold freshness under 5 minutes; automated DR test every quarter. **Q200 — What would you validate and what defects would you hunt?** **A200** — Ticks and streaming: dedupe by event_id; out-of-order and late ticks; zero or negative prices; exchange time vs UTC; gaps in minute bars; consumer lag; at-least-once delivery creating duplicates; schema compatibility on every producer change. Reference data and calendars: trading calendar and half-days; symbol changes mapped to the same security; ISIN format; an FX rate for every date used. Corporate actions: adjusted price equals raw price divided by the cumulative factor; no jump at the split date; dividends applied once; a correction reprocesses only the right date range; Redshift, Athena caches and Redis refreshed afterwards. Cross-vendor reconciliation: closing prices from two vendors within tolerance; flag outliers. Serving consistency: Gold vs Aurora vs Redshift vs Athena counts and aggregates; cache invalidated after backfill; read-replica lag. DLQ: retry counter never exceeds 3; quarantine volume alerts; replay works without duplicates. Reports: right numbers and date; one email per run (idempotent); bounces and complaints handled through SNS. Orchestration: daily DAG order; backfill; safe rerun; time-zone-aware schedule. Compliance: Object Lock on Bronze; lifecycle; retention by jurisdiction; no cross-border replication where restricted; access reviews; market-data licence usage limits. Resilience: DR drill that promotes the replica, restores consumers and measures RPO and RTO against the targets. CI/CD: incompatible schema change is blocked; Terraform plan reviewed; rollback tested. **Q201 — What are the operational risks in the diagram?** **A201** — Market-data licensing and usage; PII in news and sentiment; schema change management; data residency and sovereignty; dead-letter queue review with an owner and SLA; artifact and state security; access governance and audit; EMR quota and capacity planning. **Q202 — How do you present the diagrams honestly?** **A202** — Say 'this is a reference design I worked out and tested in parts in my lab', but only if that is true, and never 'I ran this in production'. Interviewers reward trade-off reasoning, for example why Redis must be invalidated after a backfill, why rate limiting fails closed, why a schema registry gate sits in CI/CD, or why Bronze uses Object Lock. ### 6.3 AWS Service Comparisons **Q203 — S3 vs EBS vs EFS?** **A203** — S3 is object storage for durable data lake objects; EBS is block storage attached to compute; EFS is managed shared file storage. The architecture uses S3 for scalable, durable raw and curated datasets, not as a mounted block disk. **Q204 — SQS vs SNS vs EventBridge?** **A204** — SQS is a durable queue for decoupled work; SNS is pub/sub fan-out and notifications; EventBridge routes events based on rules and schedules. Use each for its delivery semantics: queue work, fan out alerts, or trigger workflows from events/schedules. **Q205 — Kinesis vs MSK/Kafka?** **A205** — Both support streaming. Kinesis is an AWS-managed streaming service with AWS-native shard/consumer semantics; MSK manages Apache Kafka compatibility and its ecosystem. Choose based on existing Kafka APIs/tools, operations, throughput, retention, consumer needs and cost. **Q206 — Lambda vs ECS/Fargate?** **A206** — Lambda suits event-driven functions with bounded execution; ECS/Fargate suits long-running containers such as persistent WebSocket collectors. The market-data design uses Fargate for persistent connections and Lambda for short REST/event-driven tasks. **Q207 — Athena vs Redshift?** **A207** — Athena runs serverless SQL queries over data in S3; Redshift is a managed analytical warehouse for repeated, concurrent and modelled analytics workloads. Use Athena for ad-hoc lake queries and Redshift when workload patterns justify a warehouse. **Q208 — Aurora vs Redshift?** **A208** — Aurora is a relational database for transactional/serving patterns; Redshift is an analytical warehouse. Keep transactional serving separate from large scans and aggregations where possible. **Q209 — IAM vs Lake Formation?** **A209** — IAM authorises AWS API and resource access; Lake Formation adds fine-grained data-lake permissions for catalogued databases/tables/columns/rows where supported. Effective access may require both layers to permit the operation. **Q210 — KMS vs Secrets Manager?** **A210** — KMS manages cryptographic keys and cryptographic operations; Secrets Manager stores, retrieves and rotates secrets such as credentials. Encrypting a secret and granting access to retrieve it are separate controls. **Q211 — CloudWatch vs CloudTrail?** **A211** — CloudWatch monitors logs, metrics, dashboards and alarms; CloudTrail records AWS API activity for audit and investigation. Use CloudWatch to detect operational symptoms and CloudTrail to investigate who/what called an API. **Q212 — Glue vs EMR Serverless?** **A212** — Both can run Spark workloads. Glue is a managed data integration service with catalog and ETL-oriented features; EMR Serverless provides serverless runtime for Spark/Hive applications with more EMR ecosystem flexibility. Choose based on job requirements, dependencies, runtime controls, governance integration and operational cost. **Q213 — Step Functions vs MWAA/Airflow?** **A213** — Step Functions is a managed state-machine workflow service well suited to service orchestration and explicit retry/catch paths; MWAA runs managed Airflow and suits DAG-centric workflows, scheduling, backfills and Python ecosystem integration. Avoid two orchestrators owning the same workflow without a clear boundary. **Q214 — Parquet vs CSV vs JSON?** **A214** — CSV is simple but weakly typed; JSON supports nested semi-structured data but can be verbose; Parquet is columnar and typically efficient for analytical scans. Preserve original vendor payloads when needed, then standardise curated analytics data to a suitable columnar format. **Q215 — Delta Lake vs Apache Iceberg?** **A215** — Both provide table-layer capabilities on data-lake files, including transactional metadata and schema/partition evolution features, with differences in ecosystem support and operations. Choose the format supported by your engines, catalog, concurrency needs, governance and operational tooling. --- ## SECTION 7 — AZURE DATA FACTORY QUESTIONS **Q216 — What is ADF?** **A216** — Azure Data Factory is a cloud data integration service. Pipelines contain activities; datasets describe data structures/locations; linked services define connection information; Integration Runtime provides data movement, data-flow compute or activity dispatch. Triggers schedule or event-start pipelines. **Q217 — What is the difference between pipeline, activity, dataset, linked service and Integration Runtime?** **A217** — Pipeline orchestrates; activity is a step; dataset describes data shape/location; linked service defines connection; IR provides movement/compute/network execution. **Q218 — Which Integration Runtime would you choose?** **A218** — Azure IR for supported cloud data movement/data flows and dispatch; Self-hosted IR for on-premises or network-isolated sources that require a runtime in that network; Azure-SSIS IR to execute SSIS packages in Azure. Confirm network path, region, throughput, credentials and private endpoint support. **Q219 — ADF vs Databricks: why use both?** **A219** — Use ADF for orchestration and data movement and Databricks for distributed Spark transformations/lakehouse processing. This separates workflow control from compute while retaining run status and dependencies in ADF. **Q220 — How do you implement a safe watermark in ADF?** **A220** — Read the last committed watermark, extract an overlap/bounded delta, deduplicate and write idempotently, validate the output, then commit the new watermark only after successful publication. Persist the watermark in a controlled metadata store. **Q221 — Tumbling-window trigger vs schedule trigger?** **A221** — Tumbling windows represent fixed, contiguous time windows with dependency/retry semantics useful for windowed processing; schedule triggers run on a calendar schedule without the same window model. Choose based on whether each time slice must be tracked and recovered independently. **Q222 — How do you pass secrets to ADF?** **A222** — Use managed identity and a secret store such as Azure Key Vault where supported, grant least privilege and avoid embedding credentials in pipeline JSON or logs. **Q223 — How do you handle schema drift in Mapping Data Flows?** **A223** — Decide whether unknown columns may flow through or whether the contract is strict. Use explicit mappings for critical datasets, capture schema changes, test downstream compatibility and block breaking changes. **Q224 — How do you troubleshoot a slow ADF pipeline?** **A224** — Use activity run details, queue/wait time, source/sink throughput, Integration Runtime capacity, partitioning, concurrency and source throttling. Separate orchestration delay from actual copy/compute time. **Q225 — How do you monitor ADF data correctness?** **A225** — Record rows read/copied/skipped, duration and throughput, but also run source-target counts, aggregates, key checks and freshness validation. Operational success is not the same as business correctness. **Q226 — How do you validate an ADF pipeline?** **A226** — Monitoring: Check rowsRead, rowsCopied, rowsSkipped, and throughput in the Monitor tab. Fault Tolerance: Configure fault tolerance to skip incompatible rows and log them to a storage account. Data Consistency: Enable "Data consistency verification" in Copy Activity. Triggers: Validate tumbling window triggers and event-based triggers. Retry Policy: Set retry counts and intervals for transient failures. **Q227 — Copy activity succeeded but target is incomplete. What do you inspect?** **A227** — Check source query/filter and watermark, source snapshot timing, mapping/schema drift, sink write mode, partition/file counts, rejected rows, IR connectivity and parallelism, then compare counts and aggregates by partition. Add a post-copy validation activity and block downstream publication if critical reconciliation fails. **Q228 — Self-hosted IR is offline. What is your response?** **A228** — Check service status, host CPU/memory/disk, network egress and DNS, firewall/proxy, credentials/certificates, version/update state and connectivity to both source and sink. If availability is critical, design a supported multi-node IR deployment and test node failover. **Q229 — How do you deploy ADF safely?** **A229** — Keep development in Git, validate pipeline JSON and linked-service parameters, use environment-specific configuration and Key Vault, deploy through CI/CD to test then production, validate triggers/IR/permissions, run smoke/reconciliation tests and keep a rollback path. Never copy production secrets into source control. **Q230 — What is the AWS-to-Azure mapping?** **A230** — | Need | AWS example | Azure example | |---|---|---| | Orchestration | Step Functions / EventBridge | ADF pipeline / trigger | | Batch extraction | Glue connectors / Lambda | ADF Copy activity + IR | | Distributed transforms | Glue Spark / EMR Serverless | Azure Databricks / Mapping Data Flows | | Object lake | S3 | ADLS Gen2 / Blob | | Metadata catalog | Glue Data Catalog | Purview + platform catalog | | Secrets/identity | IAM roles + Secrets Manager | Managed identity + Key Vault | | Governance | Lake Formation + IAM | RBAC/ACLs + Purview | | Monitoring | CloudWatch + SNS | Azure Monitor + Log Analytics | | Warehouse/serving | Redshift/Aurora | Synapse / Azure SQL / Databricks SQL | | BI | QuickSight | Power BI | | CI/CD | CodePipeline/GitHub Actions + IaC | Git integration + ADF publish/CI-CD | --- ## SECTION 8 — ETL, DATA WAREHOUSE & MIGRATION CONCEPTS **Q231 — ETL vs ELT?** **A231** — ETL transforms before loading; ELT loads raw data first and transforms inside the warehouse or lakehouse using its compute (typical on Databricks, Snowflake, BigQuery). **Q232 — What is reconciliation?** **A232** — Proving the target equals the source, or the expected transformation of it, using counts, sums, keys and row-level comparison with documented tolerances. **Q233 — Full vs incremental vs CDC?** **A233** — Full reloads everything; incremental loads new or changed rows using a watermark; CDC reads the source change log and captures inserts, updates and deletes. **Q234 — Why a staging layer?** **A234** — It lands data as-is, isolates sources from transformations, and supports reprocessing and audit. **Q235 — Fact vs dimension?** **A235** — Facts are measurable events at a declared grain (transactional, periodic snapshot, accumulating snapshot, factless); dimensions give descriptive context. **Q236 — Star vs snowflake?** **A236** — Star has denormalised dimensions around the fact (fewer joins, faster reads); snowflake normalises dimensions (less redundancy, more joins). **Q237 — Surrogate vs natural key?** **A237** — A surrogate key is system-generated and independent of the source, required for SCD2 history; the natural key is the business key. **Q238 — Conformed and degenerate dimensions?** **A238** — Conformed: shared across facts with one meaning (date, security). Degenerate: an identifier kept in the fact with no dimension table (trade or invoice number). **Q239 — Late-arriving data?** **A239** — Fact before dimension: insert an inferred or Unknown member and correct it later. Late fact: use an overlap window and reprocess the affected partition. **Q240 — Smoke vs sanity vs regression?** **A240** — Smoke: the load runs and basic counts are right. Sanity: a targeted check after a fix. Regression: the full suite proves nothing else broke. **Q241 — What is idempotency?** **A241** — Running the same load twice gives the same result. **Q242 — No mapping document?** **A242** — Profile source and target, read the code, interview the BA and developers, write the rules down, get them approved, then test. **Q243 — Lineage and how do you validate it?** **A243** — Tracing a field from source through transformations to the report; validate by following sample keys end to end and checking that the catalog matches the code. **Q244 — Validating data masking?** **A244** — Sensitive columns are masked in non-production, format is preserved, no real PII remains, joins still work, masking is not reversible. **Q245 — ETL performance testing?** **A245** — Production-like volume, time per stage against the SLA, resource use, concurrency, and a scale test at about twice the volume. **Q246 — What to automate?** **A246** — Repeatable regression, reconciliation, DQ rules and smoke checks after every load. Not one-off exploration or fast-changing, unclear requirements. **Q247 — Time is short, how do you prioritise testing?** **A247** — Go risk-based: critical financial columns, high-volume tables and recent changes first; counts and aggregates everywhere; deep checks where risk is highest. **Q248 — What are the six data quality dimensions?** **A248** — Accuracy, completeness, consistency, timeliness, validity, uniqueness. **Q249 — How do you choose test data?** **A249** — Masked production subset + synthetic edge cases (NULLs, duplicates, boundary values, future dates, negative amounts). **Q250 — How do you prioritise defects?** **A250** — Severity = technical impact (S1–S4). Priority = business urgency. A cosmetic typo in CEO report = low severity, high priority. **Q251 — How do you test a data migration?** **A251** — Before: profile source, type mapping, volumes. During: per-batch counts and sums. After: row counts, aggregates, row-hash, referential integrity, UAT. **Q252 — How do you validate a BI report?** **A252** — Rebuild the number in SQL with same filters and grain. Compare total, then drill by dimension. Check DAX filter context, relationships, refresh. **Q253 — How do you test ETL performance?** **A253** — Production volume, time per stage vs SLA, resource use, concurrency, scale test at 2× volume. **Q254 — How do you mask PII?** **A254** — Tokenize, hash, or redact. Preserve format. No real PII in non-prod. Test masking is not reversible. **Q255 — How do you test security?** **A255** — IAM least privilege, Lake Formation column/row-level, RLS in BI, KMS key access, Secrets Manager rotation, audit logs. **Q256 — How do you test DR?** **A256** — Quarterly drill: promote replica, restore consumers, measure RPO/RTO against targets (streaming RPO < 5 min, platform RTO < 1 hour). **Q257 — What is a data contract?** **A257** — Schema + semantics + SLA + ownership + versioning. Enforced in CI/CD schema gate. **Q258 — How do you validate a Delta table before/after a load?** **A258** — DESCRIBE HISTORY, VERSION AS OF, exceptAll between current and previous version. **Q259 — How do you handle late-arriving facts?** **A259** — Insert Unknown member (-1), reprocess affected partition, update fact later. **Q260 — How do you test a quarantine pattern?** **A260** — Inject bad rows; verify loaded + rejected = source; verify SNS alert; verify no bad data in Gold. **Q261 — What are the ETL testing types?** **A261** — Source profiling, metadata/schema, completeness, transformation, integrity, duplicates, data quality, incremental/CDC, SCD, error handling, restart/idempotency, performance/volume, security/masking, reporting, regression. **Q262 — How do you choose the comparison approach by data size?** **A262** — Small tables: full row-level comparison. Medium tables: aggregates plus a row hash on every row, then drill into the differences. Very large tables: partition-wise aggregates and hash buckets, row-level comparison only on partitions that mismatch, sampling for deep checks. Always: full counts and key checks on everything; deep checks for high-risk columns (amounts, dates, keys). **Q263 — What is the test strategy?** **A263** — (1) Understand: requirements, mapping document, data model, sources, volumes, SLAs, risks. (2) Plan: scope and levels, environments and test data, entry/exit criteria, tools, roles, schedule, risks. (3) Design: scenarios from every mapping row and business rule; a traceability matrix. (4) Execute: smoke → functional → negative → incremental → regression → performance; automate what repeats. (5) Report: daily status, pass %, defect trend, coverage, open risks, sign-off with evidence. (6) Improve: every escaped defect becomes a new regression test. **Q264 — What are the migration testing steps?** **A264** — Before: inventory of objects; profile the source; data-type mapping; cleansing rules; volumes and cut-over window; rollback plan; a representative test subset. During: dress-rehearsal migrations; per-batch reconciliation of counts and sums; reject and error logs; load performance; restart after failure. After: row counts per table; aggregates; row-hash or row-level comparison; referential integrity and constraints; duplicates; business-rule spot checks with business users; sequences and identity values; indexes and statistics; permissions; downstream reports and interfaces; UAT sign-off; parallel run of old and new for a few cycles. **Q265 — What are common migration defects?** **A265** — Truncation (VARCHAR length), precision and rounding, date and time-zone shifts, encoding, NULL vs blank, case and collation, trailing spaces, orphan rows from load order, lost audit columns, wrong defaults, sequences not reset, partial loads. **Q266 — What is the defect report template?** **A266** ```text Title: [Table.Column] short description of the mismatch Environment / build / run id: Mapping or requirement ref: Steps and SQL used: Expected vs actual: Evidence: counts, 5 sample keys, SQL file, screenshot Impact: downstream reports, financial figures affected Suspected cause category: source data / mapping / code / environment / requirement gap Severity / priority: ``` **Q267 — What is the example defect title?** **A267** — FACT_TRADES.amount_usd differs from rule R12 for [N] EUR trades dated [date]: FX rate taken from load date instead of trade date. Evidence: query returns [N] rows, total difference [amount]. **Q268 — What are defect metrics people ask about?** **A268** — Pass %, defect density, defect leakage (found after release), defect removal efficiency, coverage of mapping rows, automation %. **Q269 — Agile testing and documentation?** **A269** — In-sprint testing: write scenarios during refinement; test as soon as a story is in the test environment; the definition of done includes passing automated reconciliation. Shift left: review the mapping document for ambiguity before code exists; unit tests for transformations; SQL and code review. Documents you maintain: test strategy and plan, mapping with traceability matrix, test cases and SQL scripts in Git, automation README, runbook, test summary report. **Q270 — What are the design rationales (why choose these approaches)?** **A270** — Why hash comparison: cheap for wide tables; drill down only on mismatched hashes. Why EXCEPT ALL over EXCEPT: preserves duplicates so duplicate-count mismatches are visible. Why LEFT ANTI JOIN over NOT IN: NULL-safe and faster on large data. Why salting: balances skewed joins. Why MERGE over INSERT: idempotent retries. Why quarantine: bad records kept; good records continue; loaded + rejected = source. Why watermark: incremental loads; overlap window for late data. Why Delta/Iceberg over plain Parquet: ACID, time travel, schema evolution, OPTIMIZE/ZORDER. Why Step Functions vs MWAA: serverless/simple vs complex DAGs/Airflow standard. Why Great Expectations: declarative DQ rules; pipeline gate; data docs. Why VPC endpoints: keep traffic in AWS network; avoid NAT cost; security. Why Lake Formation over IAM: fine-grained access + audit. --- ## SECTION 9 — DATA WAREHOUSE & SCD **Q271 — What is SCD Type 1?** **A271** — Overwrite the prior value; no history row; row count unchanged. **Q272 — What is SCD Type 2?** **A272** — Preserves history by inserting a new dimension version when tracked attributes change, end-dating the prior version, and assigning a new surrogate key. Test exactly one current row per natural key, non-overlapping effective periods, no unintended gaps, correct fact-to-version joins and idempotent reruns. **Q273 — What is SCD Type 3?** **A273** — Stores limited prior-state columns (previous value column populated on change). **Q274 — What SCD test scenarios should you quote?** **A274** — Type 1 (overwrite): value updated, no history row, row count unchanged. Type 2 (history): changed attribute creates a new version with a new surrogate key; old version is end-dated; unchanged rows are untouched; facts keep pointing at the version valid on the transaction date. Type 3 (previous value column): previous-value column populated on change. Also test: brand-new key, two changes on the same day, the same file reprocessed (no duplicate versions), late-arriving change, and a delete or inactive flag. **Q275 — What is the grain check?** **A275** — One row per declared grain: GROUP BY account_key, security_key, date_key HAVING COUNT(*) > 1. **Q276 — What is a late-arriving dimension?** **A276** — Facts pointing at the Unknown member (-1) must be tracked and re-pointed later. **Q277 — What is a factless fact?** **A277** — A fact table with no measures, only foreign keys, representing an event or relationship (e.g., attendance). **Q278 — What is a conformed dimension?** **A278** — Shared across facts with one meaning (date, security). **Q279 — What is a degenerate dimension?** **A279** — An identifier kept in the fact with no dimension table (trade or invoice number). **Q280 — What is the OLTP vs OLAP difference?** **A280** — OLTP supports frequent transactional reads/writes (Aurora); OLAP supports analytical scans, aggregations and large historical queries (Redshift). **Q281 — When might you choose Aurora, Redshift, Athena or ElastiCache?** **A281** — Aurora for relational serving, Redshift for warehouse analytics, Athena for ad-hoc S3 queries, and ElastiCache for low-latency hot data. **Q282 — What is read replica lag and why does it matter?** **A282** — Replicas may trail the writer; consumers can read stale data, so monitor lag and define consistency requirements. **Q283 — What is cache invalidation?** **A283** — Removing or refreshing cached values when the source of truth changes so clients do not see stale results. **Q284 — How do you publish consistent data to multiple serving targets?** **A284** — Promote a validated version with a common batch/version ID, track each target's completion, make downstream freshness explicit and reconcile before announcing completion. --- ## SECTION 10 — KAFKA / MSK QUESTIONS **Q285 — What are a Kafka topic, partition, offset and consumer group?** **A285** — A topic is a stream category; partitions are ordered logs; offsets identify positions; a consumer group shares partitions among consumers. **Q286 — Why use a schema registry?** **A286** — To version and validate event schemas and enforce compatibility rules before producers and consumers drift apart. **Q287 — How do partition keys affect ordering and throughput?** **A287** — Ordering is guaranteed within a partition, not globally; a hot key can create skew, while too many partitions increase operational overhead. **Q288 — What is consumer lag and how do you troubleshoot it?** **A288** — Lag is the gap between produced and consumed offsets; check input rate, processing time, failures, partition assignment and downstream sink latency. **Q289 — Can Kafka guarantee end-to-end exactly-once delivery?** **A289** — Not automatically. Guarantees depend on producer, processing and sink transaction semantics; design idempotent sinks and validate replay behavior. **Q290 — A schema adds a required field and old consumers fail. What went wrong?** **A290** — The change was incompatible with the consumer contract. Enforce registry compatibility, use safe additive evolution/defaults and roll out consumers/producers in a compatible order. **Q291 — One partition is overloaded while others are idle. How do you fix it?** **A291** — Profile key distribution, choose a better partition key, consider key salting where semantics allow, scale consumers within partition limits and review partition count. **Q292 — What should a Kafka/MSK recovery test cover?** **A292** — Broker/zone failure, producer retry, consumer restart, offset recovery, duplicate processing, lag catch-up, retention/replay, schema rejection and DLQ behavior. **Q293 — What are Kafka basics?** **A293** — Topics are split into partitions; producers write, consumer groups read using offsets; ordering holds within a partition; replication and retention; KRaft replaces ZooKeeper. Test for duplicates (at-least-once), consumer lag, ordering, schema compatibility and dead-letter handling. **Q294 — What is the difference between at-least-once, at-most-once and exactly-once?** **A294** — At-most-once may lose a message but avoids redelivery; at-least-once retries and may duplicate; exactly-once requires coordinated processing/commit semantics and is not automatically end-to-end across arbitrary sinks. Practical design uses durable offsets/checkpoints plus idempotent writes and reconciliation. --- ## SECTION 11 — ORCHESTRATION & SCHEDULERS **Q295 — What is the purpose of workflow orchestration?** **A295** — Coordinate dependencies, schedules, parameters, retries, timeouts, branching, status and notifications across tasks. **Q296 — Step Functions vs MWAA/Airflow?** **A296** — Step Functions suits service-oriented state workflows; MWAA suits DAG-centric scheduled data workflows. Avoid duplicating orchestration for the same workflow without a clear reason. **Q297 — Retry vs catch/error branch?** **A297** — Retry handles transient attempts; catch routes exhausted or non-retryable failures to recovery, compensation or notification logic. **Q298 — How do you prevent overlapping daily runs?** **A298** — Use scheduler concurrency/dependencies, run locks or partition-level coordination, and idempotent writes; define how late previous runs affect the next run. **Q299 — A DAG is green but a downstream table is stale. What should be investigated?** **A299** — Check skipped/short-circuited tasks, wrong run parameters, empty input, stale watermark, asynchronous task completion and whether the DAG validates freshness before success. **Q300 — How do you set retries for a database timeout vs schema validation error?** **A300** — Retry transient timeouts with bounded backoff; fail fast for deterministic schema errors and quarantine/alert rather than repeat the same invalid input. **Q301 — How would you recover from a partially completed fan-out workflow?** **A301** — Track per-source/task state, retry only failed branches, use idempotent task outputs and run a final reconciliation gate before publication. **Q302 — What must an orchestration run log contain?** **A302** — Run/correlation ID, parameters, code version, source partitions, task attempts, timestamps, output counts, DQ results, error details and final publication status. **Q303 — Airflow DAG example.** **A303** — Check exchange calendar → determine completed sessions → fetch missing data → run batch ETL → data quality validation → update serving layer → invalidate cache → generate reports → idempotent email. Task states: success, failed, upstream_failed, skipped, up_for_retry. **Q304 — Autosys basics?** **A304** — Jobs defined in JIL (command, box and file-watcher job types); attributes such as command, machine, condition, date_conditions, days_of_week, start_times, box_name, std_out_file, n_retrys, alarm_if_fail. Statuses: SU, FA, RU, IN, AC, ST, OH, OI, TE. Commands: autorep -J jobname, sendevent -E FORCE_STARTJOB, JOB_ON_HOLD, JOB_OFF_HOLD, JOB_ON_ICE, JOB_OFF_ICE, KILLJOB, CHANGE_STATUS -s SUCCESS. **Q305 — Control-M basics?** **A305** — Jobs live in folders; in and out conditions control dependencies; resources limit concurrency; calendars define run days. Statuses: Wait Condition, Executing, Ended OK, Ended Not OK. Actions: Hold, Free, Rerun, Force OK, Kill. On and Do statements react to exit code or output; SLA management raises alerts on late jobs. **Q306 — How would you validate a nightly batch?** **A306** — Before the run: inputs arrived, control-file counts present, previous run complete, parameters correct. During: job order, durations against baseline, logs and exit codes. After: counts and reconciliation, DQ gate result, reject counts, SLA met, downstream jobs triggered. Failure drills: kill a job midway, rerun, confirm no duplicates and no gaps; late, empty, duplicate and schema-changed files; confirm the alert reaches the right people. **Q307 — What is the batch job validation checklist?** **A307** — Order and dependencies, calendars and holidays, file-watcher start conditions, time windows and SLA. Retry and restart behaviour, concurrency limits, alerts to the right group, exit codes, logs. Rerun safety (idempotent), failure path, manual hold and force-success handling, environment parameters, data hand-off between jobs. --- ## SECTION 12 — SECURITY, GOVERNANCE & COMPLIANCE **Q308 — What is least privilege?** **A308** — Grant only the actions and resources needed for a role, with periodic review and separation of duties. **Q309 — Why should credentials not be stored in code or logs?** **A309** — They can leak through repositories, build logs and diagnostics; use a secret manager and short-lived role-based identity where possible. **Q310 — Why can KMS encryption cause AccessDenied even when S3 permissions look correct?** **A310** — The principal may lack KMS key-policy/grant permissions or the key policy may not trust that role; diagnose both data and key authorization. **Q311 — What are private endpoints and why use them?** **A311** — They provide private network access to supported services, reducing exposure to public endpoints; DNS, routes, policies and security groups still need validation. **Q312 — A Glue job suddenly gets AccessDenied after deployment. How do you troubleshoot?** **A312** — Compare execution role, resource policy, KMS key policy, Lake Formation grants, secret access, environment parameters and CloudTrail denial event. **Q313 — How do you protect market/customer sensitive data in logs and DLQs?** **A313** — Minimise payload retention, mask/tokenise sensitive fields, restrict access, encrypt, set retention limits and audit access. **Q314 — How do you enforce data residency?** **A314** — Classify data, define permitted regions, constrain storage/replication/processing through policies, review vendor paths and audit location/retention. **Q315 — What is the difference between IAM authorization and Lake Formation permissions?** **A315** — IAM controls AWS API/service access; Lake Formation adds fine-grained data-lake/catalog permissions. Both layers may need to allow the operation. **Q316 — What is the difference between IAM roles and policies?** **A316** — Roles are identities assumed by services/users; policies are documents defining allowed/denied actions on resources. **Q317 — What is PII and how do you mask it?** **A317** — Personally Identifiable Information. Mask in non-prod by tokenizing, hashing or redacting; preserve format; ensure no real PII remains; verify joins still work; verify masking is not reversible. **Q318 — How do you encrypt data at rest and in transit?** **A318** — At rest: KMS-managed keys for S3, MSK, RDS, Redshift. In transit: TLS for all connections, IAM auth for MSK, private endpoints. **Q319 — How do you manage cross-account access securely?** **A319** — Use IAM roles with trust policies, external IDs where applicable, Lake Formation cross-account grants, and least-privilege resource policies. **Q320 — How do you audit access to sensitive data?** **A320** — CloudTrail for API activity, Lake Formation audit logs, S3 access logs, CloudWatch Logs, and regular access reviews. **Q321 — What is data lineage vs provenance?** **A321** — Lineage = where data came from and how it transformed. Provenance = origin and ownership. **Q322 — What is a data contract vs schema?** **A322** — Schema = structure. Contract = schema + semantics + SLA + ownership + versioning. **Q323 — How do you enforce data residency and GDPR?** **A323** — Classify data, define permitted regions, constrain storage/replication/processing through policies, review vendor paths, audit location/retention, and maintain retention by jurisdiction. --- ## SECTION 13 — CI/CD, MONITORING & OBSERVABILITY **Q324 — What should a data pipeline dashboard show?** **A324** — Success/failure, duration, freshness, row counts, rejects, retries, backlog/lag, resource utilisation and cost. **Q325 — What is the difference between a log, metric and alert?** **A325** — Logs provide event details; metrics are numeric time series; alerts evaluate conditions and notify an owner. **Q326 — What tests should run before production deployment?** **A326** — Unit and transformation tests, schema/contract tests, integration tests, reconciliation, security/IaC checks and a small smoke run. **Q327 — Why use infrastructure as code?** **A327** — It makes infrastructure repeatable, reviewable and version-controlled, reducing configuration drift. **Q328 — The job failed but no alert fired. What controls do you add?** **A328** — Test alert delivery, define missing-heartbeat/freshness alarms, ensure all failure paths emit metrics/events and assign alert ownership/runbooks. **Q329 — How do you release a breaking schema change?** **A329** — Version the contract, test producer/consumer compatibility, deploy consumers in a safe order, use canary/shadow validation where possible and maintain rollback. **Q330 — How do you prevent secrets leaking through CI/CD?** **A330** — Use managed secret references/OIDC or role-based credentials, mask logs, scan repositories/artifacts, limit pipeline permissions and rotate exposed secrets. **Q331 — What does rollback mean for a data pipeline?** **A331** — Restore compatible code/config and keep or restore a known-good data version; code rollback alone cannot undo a bad data write, so data rollback/replay must be designed. **Q332 — What is Git branching strategy?** **A332** — Feature branches from develop; pull requests with review; merge to develop; release branches to main; hotfix branches for production issues. **Q333 — What is a pull request?** **A333** — A request to merge code changes, with review, automated checks and approval before merge. **Q334 — What are Jenkins pipeline stages?** **A334** — Checkout, install dependencies, run tests, build artifact, deploy to test, run integration tests, approve, deploy to prod, monitor. **Q335 — What is a build artifact?** **A335** — The output of a build (JAR, wheel, Docker image, ADF publish artifact) stored for deployment. **Q336 — CloudWatch vs SNS vs SQS?** **A336** — CloudWatch monitors metrics/logs/alarms; SNS is pub/sub notifications; SQS is a durable queue for decoupled work. **Q337 — How do you implement a schema gate in CI/CD?** **A337** — Compare the new schema with the registered version; fail the build if the change is breaking; require a compatible additive change or a migration plan. **Q338 — How do you manage infrastructure as code?** **A338** — Terraform/CloudFormation in Git, plan review, approval, apply, drift detection, rollback. **Q339 — How do you roll back a failed deployment?** **A339** — Restore the previous artifact version, revert infrastructure change, and if data was written, replay/restore from a known-good version. **Q340 — What does observability mean for data pipelines?** **A340** — Operational health plus data health. Track duration, retries, errors, backlog/lag, cost and resource saturation alongside freshness, volume, null rate, duplicate rate, schema version and reconciliation differences. Alerts should include run ID, dataset, partition, severity, owner and runbook link. **Q341 — What is a good production alert?** **A341** — Actionable, tied to user/business impact, deduplicated, routed to an owner and linked to a runbook. Avoid noisy alerts without a clear threshold or remediation path. **Q342 — What is a canary or shadow release for data pipelines?** **A342** — A canary processes a small controlled slice before broad rollout; shadow processing runs new logic alongside the current path for comparison without replacing production output. Use reconciliations and a rollback plan. **Q343 — Unit vs integration vs end-to-end test?** **A343** — Unit tests isolate a function; integration tests check interactions between components; end-to-end tests validate the whole path. Use a pyramid: many fast unit/contract tests, targeted integration tests, and a small number of realistic end-to-end tests. **Q344 — What is a data reconciliation fingerprint?** **A344** — A compact set of counts, sums, minima/maxima, distinct-key counts and optionally hashes by partition. It helps localise differences quickly but should not replace row-level investigation for financial or critical datasets. **Q345 — What is the first step when a pipeline is green but data is wrong?** **A345** — Quantify the impact and find the first boundary where source and target diverge. A successful task status proves execution, not business correctness. **Q346 — How do you write a blameless postmortem?** **A346** — State impact/timeline, detection gap, technical and contributing causes, recovery, what went well, and tracked preventive actions with owners/dates. Focus on system conditions rather than individual blame. **Q347 — How do you test disaster recovery?** **A347** — Simulate the defined failure, invoke documented failover, validate data freshness and correctness, measure achieved RPO/RTO, test downstream consumers and capture gaps. Replication configuration alone is not proof of recoverability. --- ## SECTION 14 — DISASTER RECOVERY, CAPACITY & COST **Q348 — What are RPO and RTO?** **A348** — RPO is the acceptable amount of data loss measured in time; RTO is the target time to restore service. **Q349 — Why monitor cost as well as job success?** **A349** — A technically successful job can be inefficient, unexpectedly expensive or exceed budget due to retries, scans, idle compute or data growth. **Q350 — How do you estimate capacity?** **A350** — Measure event/file rates, peak bursts, data size, transformation complexity, concurrency, retention and SLA; load-test with headroom and monitor saturation. **Q351 — When is multi-region DR justified?** **A351** — When business RPO/RTO, resilience or regulatory requirements justify replication and operational complexity; define recovery objectives first. **Q352 — A region fails. What steps prove that failover worked?** **A352** — Declare incident, promote/route to recovery region, validate dependencies/identity/catalog/offsets, reconcile latest data against RPO, run smoke tests and measure elapsed RTO. **Q353 — Replication is enabled, so are we DR-ready?** **A353** — No. Replication lag, missing metadata/secrets, DNS, quotas and failover procedures can still fail; perform timed recovery exercises. **Q354 — How do you reduce Athena/S3 query cost?** **A354** — Use columnar formats, compression, partition pruning, sensible file sizes, projected partitions where appropriate and avoid scanning unused columns/data. **Q355 — EMR/Glue costs double after a release. How do you investigate?** **A355** — Compare input volume, run count/retries, executor/worker sizing, shuffle/spill, file count, concurrency, job duration and query scans; fix root cause and add budget alarms. **Q356 — What is vertical vs horizontal scaling?** **A356** — Vertical scaling increases the capacity of a node; horizontal scaling adds nodes/parallel workers. Distributed Spark and Kafka can scale horizontally but still face skew, coordination and per-partition limits. **Q357 — Why can more Spark executors make a job slower?** **A357** — More workers can increase scheduling, shuffle, small-file and coordination overhead; the bottleneck may be a single skewed key, source rate limit or sink throughput rather than CPU capacity. **Q358 — What is a cost guardrail?** **A358** — Budgets, alarms, quotas, concurrency limits, tagging and per-job cost/usage metrics. Use them to catch runaway retries, unbounded scans, oversized clusters and unnecessary duplicate serving systems. **Q359 — What is the difference between Multi-AZ and multi-region?** **A359** — Multi-AZ protects against some zone-level failures within a region; multi-region supports recovery from a broader regional outage but needs replication, routing, data-residency decisions and tested failover. **Q360 — How do you design a DR plan with RPO < 5 min and RTO < 1 hour?** **A360** — Replicate MSK across regions, replicate S3/catalog/data, maintain standby or promoted serving systems, automate failover steps, run quarterly game days, measure achieved RPO/RTO, and document runbooks with owners. **Q361 — When should you not add streaming?** **A361** — When the business SLA can be met by batch and there is no clear need for lower latency. Streaming adds state, checkpoint, lag, replay and operational complexity. --- ## SECTION 15 — FINANCIAL / CAPITAL MARKETS DOMAIN **Q362 — What is NAV?** **A362** — Fund assets minus liabilities, divided by units outstanding (price per unit). **Q363 — What is AUM?** **A363** — Assets under management. **Q364 — What is a position or holding?** **A364** — Quantity of a security held on a date. Market value = quantity × price × FX rate. **Q365 — What is the trade lifecycle?** **A365** — Order → execution → confirmation → clearing → settlement. US equities settle on T+1 since 28 May 2024. **Q366 — What are corporate actions?** **A366** — Splits, dividends, mergers, symbol changes; adjusted vs unadjusted prices. **Q367 — What is OHLCV?** **A367** — Open, high, low, close and volume bars. **Q368 — What is reference data?** **A368** — Security master (ISIN, CUSIP, SEDOL, ticker), FX rates, holiday calendars, index constituents. **Q369 — What are performance and risk measures?** **A369** — Returns vs benchmark; standard deviation, beta, Sharpe ratio, VaR, tracking error. **Q370 — What is reconciliation in finance?** **A370** — Trades vs settlements, positions vs custodian, cash vs bank, general ledger vs sub-ledger. **Q371 — What are compliance words to know?** **A371** — Audit trail, lineage, retention, PII, KYC and AML, GDPR, data residency. **Q372 — What is mortgage and wealth vocabulary?** **A372** — LTV (loan to value), DTI (debt to income), amortisation, delinquency buckets (30, 60, 90 days past due), origination vs servicing; portfolio, household, advisor, fees. **Q373 — What are financial validation rules you can quote?** **A373** — Market value = quantity × price × FX rate, within a rounding tolerance. Position roll-forward: opening quantity + buys - sells +/- corporate actions = closing quantity. Portfolio weights sum to 100% within tolerance. NAV per unit = (assets - liabilities) / units outstanding. Trade date is not after settlement date; settlement falls on a business day under the market's T+1 or T+2 rule. No duplicate trade IDs; every trade has a valid account and security; status transitions are legal. Prices are positive; OHLC is consistent; volume is not negative; no unexplained jump above [20]% (a missed split); no gaps on trading days. Adjusted price = raw price divided by the cumulative split factor. An FX rate exists for every currency and date used. Cross-source: vendor A vs vendor B closing prices within tolerance. **Q374 — How do you validate corporate actions?** **A374** — Validate instrument identity, action type, effective/ex-date, ratio/factor, currency and vendor agreement; test price/quantity adjustment logic and continuity around split dates. **Q375 — How do you handle currency conversion?** **A375** — Define base/quote currency, rate source, rate timestamp/date, market calendar, decimal precision and rounding. Reconcile using the same rate convention and avoid silently mixing intraday and end-of-day rates. **Q376 — What is event time vs processing time?** **A376** — Event time is when the source says the event occurred; processing time is when the pipeline handled it. Store timestamps with timezone semantics, derive market business date using the exchange calendar/timezone, and monitor lateness. **Q377 — Why are market calendars important?** **A377** — Trading sessions vary by exchange, holiday, early close and daylight-saving changes. A simple weekday filter can falsely flag valid gaps or accept missing sessions; use a governed exchange calendar. **Q378 — What is late-arriving fact vs late-arriving dimension?** **A378** — A late fact arrives after its event date; a late dimension means the descriptive entity data arrives after facts reference it. Define unknown/inferred dimension handling and reprocess affected facts when the dimension becomes available. --- ## SECTION 16 — BI / DASHBOARD VALIDATION **Q379 — How do you validate a Power BI / QuickSight dashboard?** **A379** — (1) Definition: get the measure definition and filters from the specification. (2) Rebuild the number in SQL at the same grain with the same filters; compare the total, then drill down by each dimension to find where it diverges. (3) Visual checks: slicer and filter behaviour, drill-through, default selections, sort order, number formats, blank or Unknown categories, totals vs the sum of parts. (4) Model checks: relationship cardinality and direction, duplicate keys in a dimension, inactive relationships, a proper date table. (5) DAX checks: filter context with CALCULATE, ALL and ALLEXCEPT, time intelligence (YTD, prior period), measures vs calculated columns, divide by zero with DIVIDE. (6) Refresh: scheduled refresh succeeded, the data-as-of label is right, incremental refresh behaves. (7) Security: test row-level security by role (View as role), export permissions. (8) Performance: load time, number of visuals, query folding. **Q380 — Why might a dashboard number differ from SQL?** **A380** — Different filter context, relationships, duplicate dimensions, measure definitions, timezone, refresh timing or row-level security. **Q381 — What is DAX? Write a measure for AOV.** **A381** — DAX is the formula language for Power BI. AOV = DIVIDE(SUM(orders[amount]), COUNT(orders[order_id])). **Q382 — Why should a dashboard have a freshness timestamp?** **A382** — Users need to distinguish current data from a successful but stale refresh. **Q383 — What is a semantic layer?** **A383** — A governed set of business definitions, measures, dimensions and relationships shared by reports. **Q384 — The daily report was emailed although the ETL failed. How do you prevent it?** **A384** — Make report generation depend on validated publication status and freshness; failed gates block delivery and trigger operational alerting. **Q385 — How do you test automated market summaries?** **A385** — Compare report metrics to canonical SQL, verify session/date/filter definitions, check top-N tie rules, validate empty/late data behavior and track delivery status. **Q386 — An email provider accepts a send request but recipients receive nothing. What do you check?** **A386** — Delivery/bounce events, suppression lists, identity/domain verification, quotas, spam controls and SES/SNS event logs. **Q387 — How do you safely correct a published financial report?** **A387** — Record the incident, recalculate affected periods, reconcile the corrected output, version the report, notify stakeholders and retain the original plus correction audit trail. **Q388 — How do you validate a KPI?** **A388** — Define its business formula and grain, recompute from trusted detail with the same filters, and compare values and row scope. **Q389 — How do you test row-level security in BI?** **A389** — Use the "View as role" feature, verify each role sees only permitted data, check exports, and confirm no data leakage through relationships. **Q390 — How do you validate a report after a schema change?** **A390** — Verify the semantic model refresh, check that all measures still resolve, compare totals to SQL, test drill-through and slicers, and confirm no visual is broken. --- ## SECTION 17 — CLOUD PLATFORM COMPARISONS **Q391 — AWS vs Azure service mapping.** **A391** — | Need | AWS | Azure | |---|---|---| | Orchestration | Step Functions / EventBridge | ADF pipeline / trigger | | Batch extraction | Glue / Lambda | ADF Copy + IR | | Distributed transforms | Glue Spark / EMR Serverless | Databricks / Mapping Data Flows | | Object lake | S3 | ADLS Gen2 / Blob | | Metadata catalog | Glue Data Catalog | Purview | | Secrets/identity | IAM + Secrets Manager | Managed identity + Key Vault | | Governance | Lake Formation + IAM | RBAC/ACLs + Purview | | Monitoring | CloudWatch + SNS | Azure Monitor + Log Analytics | | Warehouse | Redshift / Aurora | Synapse / Azure SQL / Databricks SQL | | BI | QuickSight | Power BI | | CI/CD | CodePipeline / GitHub Actions + IaC | Git + ADF CI/CD | **Q392 — Delta vs Iceberg?** **A392** — Both ACID, schema evolution, time travel. Delta is Spark/Databricks-centric. Iceberg is engine-neutral (Spark, Trino, Flink). **Q393 — Data lake vs warehouse vs lakehouse?** **A393** — Lake = raw files cheap. Warehouse = modelled SQL. Lakehouse = ACID tables + governance on lake. **Q394 — Parquet vs CSV vs JSON?** **A394** — CSV simple but weakly typed; JSON nested but verbose; Parquet columnar and efficient for analytical scans. **Q395 — Why Docker?** **A395** — It packages the test framework with its dependencies so it runs the same on a laptop and in Jenkins. **Q396 — What is Great Expectations?** **A396** — Declarative data quality rules, runs as a pipeline gate, produces data docs. Define expectations (e.g., expect_column_values_to_not_be_null), run on each layer, fail pipeline on critical failures. **Q397 — What is CI/CD for data?** **A397** — Git branches, pull-request review, automated tests, deploy dev to test to prod with approval and rollback, infrastructure as code. **Q398 — What is an AI-enabled DQ use case?** **A398** — (1) Anomaly detection on volume, freshness and distribution drift with learned thresholds. (2) Profiling that suggests rules. (3) LLM help to draft test cases or summarise mapping documents, always reviewed by a person and never with sensitive data in unapproved tools. (4) Governance: catalog, lineage, PII classification and tags, access policies. **Q399 — What is the honest answer on AI-enabled DQ?** **A399** — 'I know the use cases and where they fit; my hands-on work is rule-based DQ checks in PySpark and SQL.' --- ## SECTION 18 — FAILURE DOMAIN CHECKLIST (25 Domains) **Q400 — How many failure domains are there across the two architectures?** **A400** — For interview discussion, use 25 consolidated failure domains. This is a practical checklist, not a mathematical limit. Each domain needs a detection signal, a prevention control, a recovery action and an owner. | # | Failure domain | Typical symptom | First evidence/check | Prevention/recovery | |---:|---|---|---|---| | 1 | Source/API/exchange | missing or stale data | source status, session calendar, sample payload | vendor SLA, freshness check, fallback policy | | 2 | WebSocket collector/ECS | disconnects, gaps, duplicate reconnect events | connection metrics, sequence numbers, logs | heartbeat, jittered reconnect, checkpoint sequence | | 3 | REST/Lambda ingestion | throttling, timeout, bad routing | Lambda logs, status codes, duration | rate limits, retries with backoff, DLQ | | 4 | SFTP/file landing | missing, partial, corrupt file | manifest, size, checksum, header/trailer | atomic landing, checksum, completeness gate | | 5 | Vendor quotas/rate limiter | HTTP 429, delayed batches | quota table and vendor response headers | central quota manager, concurrency caps | | 6 | MSK/Kafka | lag, unavailable broker, partition hot spot | consumer lag, broker health, partition distribution | multi-AZ, replication, balanced keys, capacity tests | | 7 | Schema/contract | deserialisation failures or silent field drift | registry compatibility and schema diff | compatibility gate, versioned contracts, quarantine | | 8 | Processing/EMR Spark | failed job, OOM, skew, slow stage | driver/executor logs, Spark UI, skew profile | right-size, AQE, broadcast when safe, salting, retry | | 9 | Data transformation | wrong currency/timezone/corporate-action value | row-level expected-vs-actual sample | mapping tests, reference-data versioning, explicit timezone | | 10 | Data-quality gate | invalid data passes or valid data is rejected | DQ result counts and thresholds | severity tiers, regression tests, fail-closed for critical rules | | 11 | S3/data lake | missing object, overwrite, wrong partition | object inventory, version, manifest, partition scan | immutable Bronze, versioning/Object Lock if required, checksums | | 12 | Catalog/metadata/lineage | Athena cannot query or wrong schema | Glue Catalog table/partition and lineage logs | catalog deployment tests, schema registration and repair | | 13 | Incremental/watermark/CDC | missing late rows or duplicate loads | watermark audit, batch ID, CDC operation counts | overlap windows, dedupe, commit watermark only after success | | 14 | Idempotency/replay/backfill | rerun doubles records or overwrites history | same-batch rerun reconciliation | deterministic keys, MERGE/upsert, replay from Bronze | | 15 | Orchestration/MWAA/Step Functions | skipped dependency, stuck run, bad retry | DAG/state-machine history, task parameters | explicit dependencies, timeout, bounded retries, runbook | | 16 | Serving/Aurora/Redshift | stale or inconsistent query results | load watermark, replica lag, query plan | transaction-aware publish, health checks, right-sized workload | | 17 | Cache/ElastiCache | stale prices after correction | cache age/key/version vs source of truth | TTL, event invalidation, correction-triggered rebuild | | 18 | Analytics/reporting/SES | wrong dashboard or missing email | query result vs canonical SQL, refresh/run logs | semantic definitions, report checks, delivery monitoring | | 19 | Security/identity/secrets | AccessDenied, expired credential, exposure | CloudTrail, IAM policy simulator, secret rotation status | least privilege, role-based access, Secrets Manager/Key Vault | | 20 | Network/DNS/private endpoints | timeout although service is healthy | route/DNS/security group/VPC endpoint checks | private connectivity tests, least-open firewall rules | | 21 | Governance/compliance/residency | unauthorised access or data in wrong region | Lake Formation grants, audit, classification | data classification, policy-as-code, access reviews | | 22 | CI/CD/IaC/release | bad deployment or schema incompatible | pipeline logs, diff, environment drift | tests, plan review, approvals, canary/rollback | | 23 | Observability/alerting | failure remains unnoticed | missing heartbeat, alert delivery test | metrics/logs/traces, synthetic checks, alert ownership | | 24 | DR/replication/RPO/RTO | failover loses data or takes too long | replication lag, restore/failover exercise | tested runbooks, periodic game days, measured recovery | | 25 | Capacity/cost/SLA | throttling, queue growth, budget surprise | lag, queue length, compute utilisation, spend alarms | quotas, autoscaling, partition/compaction, budgets and right-sizing | **Q401 — How do you use the 25-domain count in an interview?** **A401** — 'I group the architecture into 25 operational failure domains for a practical review. Each domain needs a detection signal, a prevention control, a recovery action and an owner. That is not a claim that only 25 failure modes exist.' **Q402 — What are the seven places to bisect a bad ETL result?** **A402** — Source → Landing/Bronze → Transformation/Silver → Data-quality gate → Gold/serving → Report/consumer → Audit/monitoring. Compare row counts, distinct business keys, sums, null rates, freshness and sample records at each hop. The first divergence narrows the investigation. **Q403 — Production ETL failure areas (10).** **A403** — (1) Source Systems: API rate limits, SFTP drops, RDBMS connection timeouts. (2) Ingestion Layer: EventBridge missed schedules, Lambda timeout/memory limits, Glue connection failures, MSK/Kinesis throttling. (3) Bronze/Raw Layer: S3 permission issues, KMS encryption failures, Lifecycle rules deleting needed data, Partition misalignment. (4) Orchestration: Step Functions state machine failures, timeout, retry exhaustion, Dependency check failures. (5) Silver/Cleansed Layer: Join fan-outs, Deduplication logic errors, Timezone conversion bugs, Schema drift. (6) Data Quality Gate: Great Expectations false positives/negatives, Quarantine bypass, SNS alert failure. (7) Gold/Curated Layer: Aggregation errors, Athena query timeouts, Redshift load failures, QuickSight dataset refresh failures. (8) Daily Operations & Recovery: Idempotency breaks, Backfill failures, Replay failures from immutable Bronze. (9) Security & Governance: IAM role misconfigurations, Lake Formation permission issues, Secrets Manager rotation failure. (10) Scale/DR/Cost: Auto-scaling failures, Multi-region replication lag, S3 lifecycle to Glacier retrieval delays. **Q404 — Global Stock Market failure areas (16).** **A404** — (1) Data Sources (Global): WebSocket disconnects, API rate limits, Missing corporate actions. (2) Ingestion Layer: ECS Fargate task failure, DynamoDB rate limit manager fail-closed, Lambda DLQ full, Schema Registry incompatibility. (3) Streaming Layer (MSK): MSK broker failure, Consumer lag, Out-of-order ticks, At-least-once delivery creating duplicates. (4) Processing Layer (EMR): EMR Serverless memory issues, Spark Structured Streaming checkpoint corruption, Schema evolution breaking jobs. (5) Data Lake Storage: S3 Object Lock errors, Delta/Iceberg merge conflicts, Lifecycle policy misconfigurations. (6) Serving Layer: Aurora replica lag, ElastiCache Redis cache invalidation failure, Redshift staleness (SLA < 1 hour). (7) Analytics, ML & Access: QuickSight dataset refresh failure, SageMaker endpoint failure. (8) Automated Report Reports: SES email failures, Duplicate reports due to idempotency check failure. (9) Orchestration (MWAA): DAG failure, Dependency timeout, Backfill overlap. (10) CI/CD Pipeline: Schema gate bypass, Terraform deployment failure, Rollback failure. (11) Security, Compliance & Governance: IAM misconfigurations, KMS key access denied, Lake Formation permission issues. (12) Network & VPC Architecture: VPC endpoint failure, NAT Gateway failure, Cross-region replication lag. (13) High Availability & Disaster Recovery: MSK replication failure, Aurora failover failure, DR region cold start delay. (14) Data Contracts & Schemas: Avro/Protobuf schema incompatibility, EIF/index schema changes breaking downstream. (15) SLA, RTO/RPO & Operational Targets: Streaming RPO > 5 mins, Platform RTO > 1 hour, EMR concurrency limit reached. (16) Operational Design Controls & Risk Mitigation: Market Data Licensing violation, PII in News/Sentiment, Data Residency violation. --- ## SECTION 19 — SCENARIO DRILLS **Q405 — NAV/market value is too high.** **A405** — Group totals by fund/date/currency; compare every layer; inspect join fan-out and effective-dated reference data; fix join grain; rerun affected partition idempotently. **Q406 — A new API field breaks the job.** **A406** — Compare schema contract/version; decide whether additive optional field is compatible; update schema/tests; quarantine breaking payloads; replay after deployment. **Q407 — Market stream has gaps.** **A407** — Inspect sequence numbers, collector reconnect logs, consumer lag and vendor feed status; recover from source replay if available; reconcile against session calendar; alert on gap duration. **Q408 — Duplicate prices after restart.** **A408** — Inspect event IDs, checkpoint and sink semantics; deduplicate by key/event time/source sequence; make writes idempotent; test restart/replay. **Q409 — A daily report was emailed with stale data.** **A409** — Verify session completion, Gold freshness and cache invalidation; gate report generation on validated watermark; send correction/incident notice if required. **Q410 — DR test misses RPO.** **A410** — Measure replication lag at incident time; identify unreplicated state/catalog/topic offsets; change replication/failover design; rerun a timed exercise and record evidence. **Q411 — ADF is slow.** **A411** — Break duration down by queue/IR/copy/transform/sink; inspect IR capacity, throughput, parallelism, data volume, network path and sink throttling; tune only after measurement. **Q412 — A job fails only in production.** **A412** — Compare identity/permissions, secret version, network routes, schema, parameters, quotas and data volume across environments; inspect the first causal error, not just the final cascade. --- ## SECTION 20 — FINAL RAPID-FIRE DEFINITIONS **Q413 — ETL vs ELT?** **A413** — ETL transforms before loading; ELT loads raw data first and transforms in the target platform. **Q414 — CDC?** **A414** — Captures inserts/updates/deletes rather than reloading a full table. **Q415 — Data lineage?** **A415** — Records where data came from and how it was transformed. **Q416 — Data contract?** **A416** — Agreed schema, semantics, ownership, compatibility and quality expectations. **Q417 — RPO?** **A417** — Maximum tolerable data loss measured in time. **Q418 — RTO?** **A418** — Target time to restore service. **Q419 — Freshness?** **A419** — How recently the dataset reflects the source/business event. **Q420 — Backfill?** **A420** — Intentionally reprocess a historical range. **Q421 — Replay?** **A421** — Reprocess retained raw events/data; consumers must handle duplicates safely. **Q422 — Partitioning?** **A422** — Organises data by columns such as business date to reduce scans; over-partitioning creates small-file overhead. **Q423 — Compaction?** **A423** — Combines small files into larger efficient files. **Q424 — Lakehouse table format?** **A424** — Delta/Iceberg add table metadata/transactions and schema/evolution features over object storage; design features vary by engine/version. **Q425 — At-least-once delivery?** **A425** — An event can be delivered more than once; deduplication/idempotency is needed. **Q426 — Exactly-once?** **A426** — Scope the claim carefully; end-to-end semantics depend on source, processing checkpoint and sink transaction support. **Q427 — SLA?** **A427** — Service commitment/target; measure success, freshness and delivery time against it. **Q428 — Runbook?** **A428** — Actionable steps, checks, owners and escalation paths for an incident. **Q429 — Idempotency?** **A429** — Repeating the same logical operation leaves the target in the same correct state rather than duplicating or corrupting data. **Q430 — Watermark vs checkpoint?** **A430** — Watermark = incremental load boundary. Checkpoint = streaming progress store. **Q431 — Quarantine vs DLQ?** **A431** — Quarantine = bad records kept for investigation. DLQ = failed messages for retry/replay. **Q432 — Data lineage vs provenance?** **A432** — Lineage = where data came from and how it transformed. Provenance = origin and ownership. **Q433 — SCD1 vs SCD2 vs SCD3?** **A433** — SCD1 overwrite. SCD2 add new row with dates. SCD3 previous value column. **Q434 — Fact vs dimension?** **A434** — Fact = measurable event at grain. Dimension = descriptive context. **Q435 — Star vs snowflake?** **A435** — Star = denormalised dimensions. Snowflake = normalised dimensions. **Q436 — OLTP vs OLAP?** **A436** — OLTP = transactions (Aurora). OLAP = analytics (Redshift). **Q437 — What are the four phrases to close with?** **A437** — (1) "I never trust a green pipeline, only a reconciled one." (2) "I validate at three levels: structure, volume and content." (3) "I look for the first hop where the numbers diverge." (4) "Every defect I find becomes an automated check." --- ## SECTION 21 — HONEST POSITIONING & INTERVIEW DAY **Q438 — How do you position your CV experience honestly?** **A438** — Use evidence from your CV: Python, SQL, PySpark, Azure Databricks, Azure Data Factory, ADLS, AWS S3, SSIS, data quality, reconciliation, SQL Server tuning, Git/Jenkins and Power BI. Your CV lists AWS Glue/Lambda/Redshift as exposure and Kafka/Hadoop/Iceberg/Airflow as lab experience. For those components, say "I have exposure / lab experience and can explain how I would implement and test this design" unless you can defend production ownership. **Q439 — Safe answer for the architecture diagrams.** **A439** — "These are reference architectures I designed to demonstrate production controls. My project experience is in data engineering, transformation and validation; I would not claim I operated every AWS component in the diagram in production." **Q440 — Experience-length question.** **A440** — Follow the numbers in the current CV and the specific role's requirement. Do not turn all data-quality work into a claim that your formal job title was ETL Tester. **Q441 — Likely flow of the round.** **A441** — Introduction (2 to 3 minutes) → project deep-dive (10 to 15) → root-cause scenarios → SQL (15 to 20) → Python (10) → PySpark (10) → concepts and tools → your questions. **Q442 — Live-coding routine.** **A442** — (1) Repeat the problem and clarify: NULLs, duplicates, ties, data volume, which database. (2) Give the approach in one or two sentences before typing. (3) Write clean code with sensible names. (4) Test it aloud on a tiny example, including an edge case. (5) Mention complexity or performance. (6) Finish with 'here is how I would validate this result'. (7) If stuck: say what you know, solve a simpler version, think aloud, ask for a hint. Never go silent. **Q443 — Three-day practice plan.** **A443** — Day 1: say the 90-second introduction aloud three times; rehearse three root-cause stories; write the twelve SQL validation queries from memory. Day 2: the SQL interview problems and SCD2 checks; Python (compare CSV, header and trailer, retry decorator); the shell one-liners. Day 3: the PySpark toolkit and the framework blueprint; both architecture walkthroughs; a timed mock of the whole round. **Q444 — Last-hour checklist.** **A444** — Every [bracket] replaced with a real fact or removed. You can code any CV line on the spot. Two stories ready for 'a time you found the cause of a mismatch'. Laptop, microphone, internet and a notepad ready; two questions prepared for the interviewer. Remember the rhythm: what I saw → where I narrowed it down → proof → fix → prevention. **Q445 — Final interview principle.** **A445** — Do not say only "the job succeeded." Say how you prove the data is complete, accurate, fresh, secure, reproducible and ready for consumers. --- ## APPENDIX — QUICK REFERENCE TABLES **A1 — SQL Dialect Cheat Sheet** | Topic | Oracle | SQL Server | PostgreSQL | Snowflake / Redshift / BigQuery | |---|---|---|---|---| | Set difference | MINUS | EXCEPT | EXCEPT, EXCEPT ALL | EXCEPT; Snowflake/Redshift also MINUS; BigQuery EXCEPT DISTINCT | | Replace NULL | NVL, COALESCE | ISNULL, COALESCE | COALESCE | COALESCE, IFNULL, NVL | | Limit rows | FETCH FIRST n ROWS ONLY | TOP n, OFFSET FETCH | LIMIT n | LIMIT n | | Hash functions | STANDARD_HASH, ORA_HASH | HASHBYTES, CHECKSUM | MD5 | Snowflake HASH, MD5, SHA2; Redshift MD5, SHA2, FNV_HASH; BigQuery MD5, SHA256, FARM_FINGERPRINT | | Key constraints | Enforced | Enforced | Enforced | PK, FK, UNIQUE not enforced (Snowflake still enforces NOT NULL) | | Empty string | Same as NULL | Not NULL | Not NULL | Not NULL | | Regex | REGEXP_LIKE | LIKE, PATINDEX | the ~ operator | REGEXP_LIKE, ~, REGEXP_CONTAINS | | Filter on window result | Subquery | Subquery | Subquery | QUALIFY | | Metadata | ALL_TAB_COLUMNS | INFORMATION_SCHEMA.COLUMNS | INFORMATION_SCHEMA.COLUMNS | INFORMATION_SCHEMA.COLUMNS | **A2 — PySpark Read Modes** | Mode | Behaviour | |---|---| | PERMISSIVE | Keeps row, sets bad fields to NULL, stores raw line in corrupt-record column | | DROPMALFORMED | Silently drops bad rows (dangerous for a tester) | | FAILFAST | Stops at the first bad record | **A3 — Spark Narrow vs Wide Transformations** | Narrow (no shuffle) | Wide (shuffle) | |---|---| | filter, select, map, flatMap, union | groupBy, join, distinct, repartition, sortBy | **A4 — Comparison Approach by Data Size** | Size | Approach | |---|---| | Small | Full row-level comparison | | Medium | Aggregates + row hash on every row, drill into differences | | Very large | Partition-wise aggregates and hash buckets, row-level only on mismatched partitions, sampling for deep checks | | Always | Full counts and key checks on everything; deep checks for high-risk columns | **A5 — Defect Severity vs Priority** | Severity | Meaning | Priority | Meaning | |---|---|---|---| | S1 | Data loss or wrong financial figures | P1 | Fix immediately | | S2 | Major rule wrong, workaround exists | P2 | Fix in next release | | S3 | Minor issue | P3 | Fix when time permits | | S4 | Cosmetic | P4 | Low urgency | --- **END OF COMPLETE Q&A BANK — 445 questions and answers covering SQL, Python, PySpark, AWS, ADF, ETL validation, architecture failure analysis, financial domain, BI validation, security, CI/CD, DR, cost, and interview-day playbook.** --- Source file: Pasted text (2)(20261009-072419).txt # Gaps Identified — Additional Q&A Bank (Q446–Q580) After reviewing the 445-question bank against the combined uploaded files and what interviewers actually ask at XXX/TCS/Accenture/Amazon/Microsoft for ETL/Data Engineer roles, the following **135 questions are still missing**. These are grouped into new sections and follow the same Q/A pattern. --- ## SECTION 22 — SSIS (ON YOUR CV — MUST KNOW DEEPLY) **Q446 — What is SSIS?** **A446** — SQL Server Integration Services is Microsoft's ETL tool. It uses Control Flow (workflow orchestration), Data Flow (row-by-row transformation), Event Handlers (error/notification logic), and Package Configurations (environment parameters). Deployed to SSISDB catalog, scheduled via SQL Server Agent. **Q447 — Control Flow vs Data Flow?** **A447** — Control Flow orchestrates tasks and containers (Execute SQL Task, Foreach Loop, Sequence Container, Script Task). Data Flow is where rows move through sources, transformations and destinations (OLE DB Source, Lookup, Derived Column, Conditional Split, OLE DB Destination). Control Flow manages *what runs when*; Data Flow manages *how rows transform*. **Q448 — Lookup transformation: full cache vs partial cache vs no cache?** **A448** — Full cache: loads the entire reference set into memory before the data flow starts (fast, memory-heavy, best for small dimensions). Partial cache: caches rows as encountered with a configurable cache size (balances memory and speed). No cache: queries the reference for each row (slow, for very large dimensions with sparse hits). Always configure the **no-match output** to route unmatched rows to a reject table or Unknown member — never fail silently. **Q449 — How does SSIS handle errors?** **A449** — Three levels: (1) Error output on each transformation — route bad rows to a reject destination. (2) Event Handlers — OnError, OnWarning, OnPreExecute — log to a table, send email, or execute cleanup. (3) Package-level error handling — set MaximumErrorCount, transaction options (Required/Supported/NotSupported), and checkpoints for restartability. Always capture ErrorCode and ErrorColumn to identify the failing column. **Q450 — What is a checkpoint in SSIS?** **A450** — A checkpoint file records which tasks completed successfully so that on failure the package can restart from the last successful task instead of rerunning everything. Configure CheckpointUsage (Never/IfExists/Always), SaveCheckpoint, and CheckpointFileName. Critical for long-running packages. **Q451 — How do you implement SCD Type 2 in SSIS?** **A451** — Use the Slowly Changing Dimension transformation: it detects changing, fixed, and historical attributes, then routes rows to three outputs (Insert new, Update changing, Update historical). For historical: insert a new row with a new surrogate key and effective dates, and end-date the old row. Better practice: use a MERGE statement in an Execute SQL Task for performance and testability, especially at scale. **Q452 — Row Count transformation vs Row Count in a variable?** **A452** — The Row Count transformation stores the count in a package variable at the end of the data flow, which can then be compared against a source count in a subsequent task for reconciliation. This is the SSIS equivalent of source-to-target reconciliation. **Q453 — How do you parameterize an SSIS package for dev/test/prod?** **A453** — Use **Project Parameters** (for environment-specific connection strings, file paths, schedule parameters) with the **SSIS Environment** in SSISDB. Or use Package Configurations with an indirect XML/config table. Never hardcode connection strings. In CI/CD, deploy the .ispac and reference the correct environment. **Q454 — How do you test an SSIS package?** **A454** — (1) Unit: test each Data Flow with sample data and verify row counts and transformations. (2) Integration: run the full package with a controlled subset, compare source/target counts and sums. (3) Reject handling: inject bad rows and verify they land in the reject table with the correct error. (4) Restartability: kill the package midway, rerun, verify no duplicates. (5) Performance: production-volume run against SLA. (6) Configuration: verify parameters resolve correctly in each environment. --- ## SECTION 23 — DATABRICKS (ON YOUR CV — MUST KNOW DEEPLY) **Q455 — What is Databricks?** **A455** — A unified data analytics platform built on Apache Spark, with managed clusters, notebooks, Delta Lake, Unity Catalog, Workflows, and MLflow. On Azure it integrates with ADLS and ADF; on AWS it can read S3. **Q456 — Job cluster vs all-purpose cluster?** **A456** — All-purpose clusters are interactive, shared by multiple users, good for exploration. Job clusters are created for a single job run and terminate afterward — cheaper and isolated for production. Use job clusters for scheduled pipelines. **Q457 — What is Unity Catalog?** **A457** — Databricks' governance layer that provides a unified catalog across workspaces, fine-grained access control (table, column, row), lineage, audit, and data discovery. Replaces the older Hive Metastore + table ACLs model. **Q458 — What is a widget in a Databricks notebook?** **A458** — A parameter input (text, dropdown, multiselect) that lets you pass values into a notebook without hardcoding. Used with `dbutils.widgets.get("name")`. Essential for parameterized jobs and ADF integration. **Q459 — How do you run a Databricks notebook from ADF?** **A459** — Use the Databricks Notebook activity in ADF. Configure the linked service (workspace URL + access token or managed identity), the notebook path, base parameters, and the cluster to use. Monitor the run via ADF's monitor tab and Databricks job runs. **Q460 — What is a repair run in Databricks Workflows?** **A460** — When a multi-task job fails, repair run reruns only the failed tasks and their downstream dependencies, preserving successful task outputs. Saves time and cost versus rerunning the whole job. **Q461 — How do you validate a Delta table?** **A461** — `DESCRIBE HISTORY` shows every operation (who, what, when). `SELECT * FROM table VERSION AS OF 41` reads a prior version. `SELECT * FROM table TIMESTAMP AS OF '2026-10-01'` reads by time. `exceptAll` between current and prior version shows exactly what the last run changed. **Q462 — What is OPTIMIZE with ZORDER?** **A462** — OPTIMIZE compacts small files into larger ones. ZORDER co-locates related data in the same files by the specified columns, dramatically improving query performance on those columns via data skipping. Use on columns frequently used in WHERE clauses. **Q463 — What is VACUUM?** **A463** — VACUUM removes old data files no longer referenced by the Delta transaction log, freeing storage. Default retention is 7 days. Setting it lower breaks time travel beyond that window. Run VACUUM after OPTIMIZE. **Q464 — How do you handle schema evolution in Delta?** **A464** — Use `mergeSchema = true` on write to add new columns automatically. Use `overwriteSchema = true` to replace the schema (destructive). For strict contracts, fail the write if the schema differs. Always validate in Silver before writing Gold. **Q465 — What is the difference between managed and external tables in Databricks?** **A465** — Managed tables: Databricks manages both metadata and data files (default location). Dropping the table deletes the data. External tables: metadata in the catalog, data in a specified location (e.g., ADLS). Dropping the table removes metadata only — the data remains. --- ## SECTION 24 — POWER BI / DAX (ON YOUR CV — MUST KNOW DEEPLY) **Q466 — What is DAX?** **A466** — Data Analysis Expressions — the formula language for Power BI, Analysis Services, and Excel Power Pivot. Used for calculated columns (row context) and measures (filter context). **Q467 — Calculated column vs measure?** **A467** — Calculated column is computed row-by-row at refresh time and stored in the model. Measure is computed at query time based on the filter context of the visual. Measures are more flexible and memory-efficient; use calculated columns only when you need to slice/filter by the value. **Q468 — What is filter context?** **A468** — The set of filters active when a measure is evaluated — from slicers, visual axes, page filters, and relationships. CALCULATE modifies the filter context. **Q469 — What is row context?** **A469** — The current row when evaluating a calculated column or an iterator function (SUMX, AVERAGEX). Row context does not automatically flow into measures — use RELATED or CALCULATE to transition. **Q470 — CALCULATE vs CALCULATETABLE?** **A470** — CALCULATE modifies the filter context and returns a scalar. CALCULATETABLE modifies the filter context and returns a table. **Q471 — What is ALL vs ALLEXCEPT vs REMOVEFILTERS?** **A471** — ALL removes all filters from a table or column. ALLEXCEPT removes all filters except the specified columns. REMOVEFILTERS is the modern equivalent of ALL for filter removal in CALCULATE (clearer intent). **Q472 — What is time intelligence? Write YTD.** **A472** — Time intelligence functions compute over a date table: `TOTALYTD(SUM(Sales[Amount]), 'Date'[Date])`, `SAMEPERIODLASTYEAR`, `DATEADD`, `DATESYTD`. Requires a marked date table with contiguous dates. **Q473 — Write an AOV measure.** **A473** — `AOV = DIVIDE(SUM(orders[amount]), COUNT(orders[order_id]))`. DIVIDE handles divide-by-zero safely. **Q474 — What is the difference between SUM and SUMX?** **A474** — SUM aggregates a single column. SUMX iterates a table row by row and evaluates an expression per row, then sums. SUMX is needed when the expression involves multiple columns or relationships per row. **Q475 — Why can a Power BI total differ from the sum of the visible rows?** **A475** — Because the total is computed in the total's filter context, not by summing the visible rows. This happens with non-additive measures (distinct count, averages, ratios) and incorrect relationships. Always verify with a SQL rebuild at the same grain. **Q476 — What is row-level security in Power BI?** **A476** — Roles defined in the model filter which rows a user can see. Test with "View as role". Define roles in Desktop, assign users in the Service. Common pattern: filter by user principal name or region. **Q477 — What is a star schema in Power BI?** **A477** — Fact tables (measurable events) surrounded by dimension tables (descriptive context), joined by surrogate keys. Preferred over flat tables for performance, clarity and DAX simplicity. Avoid snowflake unless necessary. **Q478 — How do you optimize a slow Power BI report?** **A478** — Reduce model size (remove unused columns, use star schema), use Import mode over DirectQuery where possible, avoid calculated columns for slicers, reduce visuals per page, use aggregations, optimize DAX (avoid iterators over large tables, use variables), and enable query folding in Power Query. --- ## SECTION 25 — GIT, JENKINS & DEVOPS (ON YOUR CV) **Q479 — Git branching strategy?** **A479** — Feature branches from develop; pull requests with review and CI checks; merge to develop; release branches cut to main; hotfix branches from main merged back to develop. Tag releases. Never commit directly to main. **Q480 — merge vs rebase?** **A480** — Merge preserves history with a merge commit. Rebase rewrites history by replaying commits on top of another branch, producing a linear history. Use rebase on feature branches before merging to keep the main history clean; never rebase shared branches. **Q481 — How do you resolve a merge conflict?** **A481** — Git marks the conflicting sections with `<<<<<<<`, `=======`, `>>>>>>>`. Open the file, decide which changes to keep (or combine), remove the markers, stage the file, and complete the merge/rebase. Communicate with the other author if the intent is unclear. **Q482 — What is a pull request?** **A482** — A request to merge code from one branch to another, with review, automated checks (tests, linting, security scanning), and approval before merge. It is the primary code-review mechanism in Git-based workflows. **Q483 — What are Jenkins pipeline stages for ETL?** **A483** — Checkout → install dependencies → run unit tests → run integration tests → build artifact → deploy to test → run smoke tests → approve → deploy to prod → monitor. Use declarative pipeline with `post` blocks for notifications and artifact archiving. **Q484 — What is a Jenkins build artifact?** **A484** — The output of a build stored for deployment or traceability: a JAR, Python wheel, Docker image, or an ADF publish artifact. Archived per build for rollback. **Q485 — How do you store secrets in Jenkins?** **A485** — Use the Jenkins Credentials plugin with a secret store (or HashiCorp Vault), reference them in the pipeline via `credentials()`, mask them in logs, and never hardcode them. Rotate regularly. **Q486 — What is Docker and why use it for data pipelines?** **A486** — Docker packages an application with its dependencies into a portable image that runs identically on any host. For data pipelines, it ensures the same Python/Spark/test environment on a laptop and in CI/CD, eliminating "works on my machine" issues. **Q487 — What is infrastructure as code (IaC)?** **A487** — Managing infrastructure (VPCs, S3 buckets, IAM roles, Glue jobs, ADF pipelines) through version-controlled code (Terraform, CloudFormation, ARM templates) rather than manual console clicks. Enables review, repeatability, drift detection, and rollback. **Q488 — Terraform state: why does it matter?** **A488** — Terraform stores the current state of managed resources in a state file. This file maps code to real resources. Store it remotely (S3 + DynamoDB for locking, or Terraform Cloud), never in Git. State corruption or concurrent applies cause drift or destruction — use locking. --- ## SECTION 26 — LINUX / SHELL (JD ASKS — CV GAP) **Q489 — How do you check file permissions?** **A489** — `ls -l` shows permissions as `rwxrwxrwx` (owner/group/other). `chmod 644 file` sets read-write for owner, read-only for others. `chown user:group file` changes ownership. `umask` sets default permissions for new files. **Q490 — How do you find large files?** **A490** — `find /path -type f -size +100M` finds files larger than 100MB. `du -sh *` shows directory sizes. `df -h` shows disk usage. **Q491 — How do you monitor a running process?** **A491** — `ps aux | grep process`, `top` / `htop` for real-time, `kill -9 PID` to force-kill, `nohup command &` to run in background and survive logout. **Q492 — How do you schedule a job on Linux?** **A492** — `crontab -e` edits the cron schedule. Format: `minute hour day month weekday command`. Example: `0 2 * * * /opt/etl/validate.sh >> /var/log/validate.log 2>&1` runs at 2 AM daily. Alternatively `systemd` timers for modern systems. **Q493 — What does `set -euo pipefail` do?** **A493** — `-e` exits on any command failure. `-u` exits on unset variable reference. `-o pipefail` makes a pipeline fail if any command in it fails, not just the last. Essential for robust shell scripts. **Q494 — How do you count rows and check duplicates in a CSV?** **A494** — `wc -l file.csv` counts lines. `cut -d',' -f1 file.csv | sort | uniq -d` shows duplicate values in column 1. `awk -F',' 'NR>1 {c[$1]++} END {for (k in c) if (c[k]>1) print k, c[k]}' file.csv` shows duplicate keys with counts. **Q495 — How do you compare two files?** **A495** — `diff file1 file2` for line-by-line. `diff <(sort a.csv) <(sort b.csv)` ignores row order. `md5sum file` for checksums. For large files, `comm -3 <(sort a) <(sort b)` shows unique lines. **Q496 — How do you search logs for errors?** **A496** — `grep -c ERROR job.log` counts errors. `grep -A 5 -B 2 ERROR job.log` shows context. `tail -n 100 job.log` shows the last 100 lines. `grep ERROR job.log | awk '{print $NF}' | sort | uniq -c` counts error types. --- ## SECTION 27 — ADVANCED SQL (BEYOND THE BASICS) **Q497 — What is a recursive CTE? Write an example.** **A497** — A CTE that references itself to traverse a hierarchy. Example: employee-manager hierarchy. ```sql WITH RECURSIVE org AS ( SELECT emp_id, emp_name, manager_id, 1 AS level FROM employees WHERE manager_id IS NULL UNION ALL SELECT e.emp_id, e.emp_name, e.manager_id, o.level + 1 FROM employees e JOIN org o ON e.manager_id = o.emp_id ) SELECT * FROM org ORDER BY level, emp_name; ``` **Q498 — What is PIVOT / UNPIVOT?** **A498** — PIVOT rotates rows into columns (e.g., quarters into Q1, Q2, Q3, Q4 columns). UNPIVOT rotates columns into rows. In SQL Server: `SELECT * FROM t PIVOT (SUM(amount) FOR quarter IN ([Q1],[Q2],[Q3],[Q4])) p`. In Snowflake/BigQuery, PIVOT is supported. In standard SQL, use CASE + GROUP BY. **Q499 — What is MERGE and how does it work?** **A499** — MERGE performs INSERT, UPDATE, and DELETE in a single statement based on a match condition. Essential for idempotent upserts. ```sql MERGE INTO target t USING source s ON t.id = s.id WHEN MATCHED THEN UPDATE SET t.amount = s.amount WHEN NOT MATCHED THEN INSERT (id, amount) VALUES (s.id, s.amount) WHEN NOT MATCHED BY SOURCE THEN DELETE; ``` Caveat: MERGE is not automatically idempotent if the source has duplicate keys — dedupe source first. **Q500 — What is a window frame and why specify ROWS?** **A500** — A window frame defines which rows are included in a window function. With only ORDER BY, the default frame is RANGE, which includes all rows with the same ORDER BY value (ties share one running total). Use `ROWS BETWEEN UNBOUNDED PRECEDING AND CURRENT ROW` to get one row at a time. **Q501 — What are isolation levels?** **A501** — READ UNCOMMITTED (dirty reads possible), READ COMMITTED (no dirty reads, phantom reads possible), REPEATABLE READ (no non-repeatable reads, phantom reads possible), SERIALIZABLE (no phantoms, strictest, slowest). SNAPSHOT uses row versioning for consistency without locks. Choose based on consistency vs concurrency needs. **Q502 — What is a deadlock and how do you resolve it?** **A502** — Two transactions each hold a lock the other needs, so neither can proceed. The database kills one as the victim. Resolve by: consistent lock ordering, shorter transactions, appropriate indexes to reduce lock scope, and retry logic for the victim. **Q503 — What is query plan and how do you read it?** **A503** — The execution plan shows how the database will execute a query: scans vs seeks, join algorithms (nested loop, hash, merge), sort operations, and cost estimates. Read it to find missing indexes, non-sargable predicates, implicit conversions, and cardinality misestimates. **Q504 — What is a covering index?** **A504** — An index that includes all columns needed by a query, so the database does not need to touch the base table. In SQL Server, use INCLUDE columns to add non-key columns. Dramatically improves read performance. **Q505 — What is a histogram / statistics in SQL Server?** **A505** — Statistics store the distribution of values in a column. The query optimizer uses them to estimate cardinality and choose a plan. Stale statistics cause bad plans. Update with `UPDATE STATISTICS table` or enable AUTO_UPDATE_STATISTICS. **Q506 — CTE vs subquery vs temp table vs table variable?** **A506** — CTE: readable, usually inlined by the optimizer. Subquery: nested, harder to read. Temp table: materialized, indexable, good for reuse in large queries. Table variable: in-memory (mostly), limited statistics, good for small sets. Choose based on size, reuse, and optimizer behavior. **Q507 — What is a correlated subquery?** **A507** — A subquery that references a column from the outer query, evaluated once per outer row. Example: `SELECT * FROM employees e WHERE salary > (SELECT AVG(salary) FROM employees WHERE dept_id = e.dept_id)`. Slow on large data — often rewrite as a window function or join. **Q508 — How do you find the Nth highest salary?** **A508** — Use DENSE_RANK: `SELECT * FROM (SELECT e.*, DENSE_RANK() OVER (ORDER BY salary DESC) rnk FROM employees e) x WHERE rnk = N`. DENSE_RANK handles ties correctly (N distinct salaries, not N rows). **Q509 — How do you delete duplicate rows keeping one?** **A509** — Use ROW_NUMBER partitioned by the duplicate key, ordered by a tie-breaker, and delete rows with rn > 1: ```sql WITH d AS (SELECT ROW_NUMBER() OVER (PARTITION BY key ORDER BY load_ts DESC) rn FROM t) DELETE FROM d WHERE rn > 1; ``` For Oracle, use ROWID instead of a CTE-based delete. **Q510 — What is the difference between RANK, DENSE_RANK, and ROW_NUMBER on ties?** **A510** — For values 100, 100, 90: ROW_NUMBER = 1, 2, 3 (arbitrary tie-break); RANK = 1, 1, 3 (gaps); DENSE_RANK = 1, 1, 2 (no gaps). Use ROW_NUMBER to pick exactly one row; DENSE_RANK to find the Nth distinct value. --- ## SECTION 28 — DATA MODELING DEEP DIVE **Q511 — What is Kimball vs Inmon?** **A511** — Kimball: bottom-up, dimensional (star schema), business-process-oriented marts, faster to deliver. Inmon: top-down, normalized enterprise warehouse, then marts, more governance but slower. Most modern warehouses use Kimball-style dimensional modeling. **Q512 — What is Data Vault?** **A512** — A modeling approach with Hubs (business keys), Links (relationships), and Satellites (descriptive attributes and history). Highly auditable and scalable, good for integrating many sources. More complex than star schema — used in large enterprises. **Q513 — What is a bridge table?** **A513** — Resolves many-to-many relationships between dimensions or between a fact and a dimension. Example: a customer can belong to multiple segments, so a bridge table maps customer_key to segment_key. Weight factors may be used to allocate measures. **Q514 — What is a fact constellation / galaxy schema?** **A514** — Multiple fact tables sharing conformed dimensions. Common in enterprises with sales, inventory, and finance facts all using the same date, product, and location dimensions. **Q515 — What is a role-playing dimension?** **A515** — A single dimension used multiple times in a fact with different meanings. Example: date dimension used as order_date, ship_date, delivery_date — joined three times with different aliases. **Q516 — What is a junk dimension?** **A516** — A dimension that combines low-cardinality flags and indicators (is_gift, is_return, is_promo) into one table to avoid cluttering the fact with many small columns. **Q517 — What is a mini-dimension?** **A517** — Splits frequently changing attributes (age band, income band) into a separate dimension to avoid SCD2 explosion on the main dimension. **Q518 — What is a degenerate dimension?** **A518** — An identifier kept in the fact table with no dimension table. Example: invoice number, order number. Used for drill-through and lineage. **Q519 — What is a factless fact?** **A519** — A fact table with no measures, only foreign keys. Represents an event or relationship. Example: student attendance — student_key, course_key, date_key. **Q520 — What is a periodic snapshot fact?** **A520** — A fact table capturing the state at regular intervals (daily, monthly). Example: end-of-day account balance. Complements transactional facts. **Q521 — What is an accumulating snapshot fact?** **A521** — A fact table with multiple date keys representing the stages of a process (order_date, ship_date, delivery_date) and updated as the process progresses. Good for pipeline/fulfillment analysis. --- ## SECTION 29 — TESTING METHODOLOGY & PROCESS **Q522 — What is the difference between verification and validation?** **A522** — Verification: "Are we building the product right?" — checking against the specification (code review, unit tests). Validation: "Are we building the right product?" — checking against the user's actual need (UAT, business rule testing). **Q523 — What is a traceability matrix?** **A523** — A table mapping requirements → mapping rules → test cases → defects. Proves coverage and helps impact analysis when a requirement changes. Essential in regulated domains. **Q524 — What is entry and exit criteria?** **A524** — Entry criteria: conditions to start testing (code deployed, test data ready, environment stable, mapping approved). Exit criteria: conditions to stop testing (all critical tests pass, no open S1/S2 defects, coverage threshold met, sign-off obtained). **Q525 — What is a smoke test?** **A525** — A quick check that the pipeline runs end-to-end and basic counts are correct. Run after every deployment. If smoke fails, stop and fix before deeper testing. **Q526 — What is a sanity test?** **A526** — A targeted check after a specific fix. Narrower than a smoke test — verifies the fix works without running the full suite. **Q527 — What is a regression test?** **A527** — A test that verifies existing behavior is unchanged after a change. Run the full regression suite before every release. Every escaped defect becomes a new regression test. **Q528 — What is exploratory testing?** **A528** — Simultaneous learning, test design, and execution without pre-scripted cases. Useful for finding edge cases that scripted tests miss. Time-box it. **Q529 — What is the difference between positive and negative testing?** **A529** — Positive: valid inputs produce expected outputs. Negative: invalid inputs are rejected with the correct error. Always test both. **Q530 — What is boundary value analysis?** **A530** — Test at the boundaries of input ranges: minimum, minimum-1, minimum+1, maximum, maximum-1, maximum+1. Most defects hide at boundaries (e.g., a 0.005 rounding boundary, a 100% weight boundary). **Q531 — What is equivalence partitioning?** **A531** — Divide inputs into equivalence classes where all values behave the same. Test one value per class. Reduces test count while maintaining coverage. **Q532 — What is a test data strategy?** **A532** — Masked production subset for realistic distributions + synthetic edge cases (NULLs, duplicates, boundary values, future dates, negative amounts, unicode, very long strings). Never use real PII in non-prod. **Q533 — What is defect leakage?** **A533** — Defects found after release (by users or production monitoring). A key metric — low leakage means effective pre-release testing. Analyze every leaked defect for a missing test. **Q534 — What is defect density?** **A534** — Defects per unit of code or per function point. Used to estimate quality and predict future defect rates. Track by module to focus testing effort. **Q535 — What is defect removal efficiency (DRE)?** **A535** — (Defects found before release) / (Total defects found before + after release). Measures how well testing catches defects. Target > 90% for critical systems. **Q536 — What is shift-left testing?** **A536** — Moving testing earlier in the development cycle: review mapping documents before code, write unit tests with the code, test in the developer's environment, automate checks in CI. Cheaper to fix defects early. --- ## SECTION 30 — BEHAVIORAL / STAR QUESTIONS **Q537 — Tell me about a time you had a conflict with a teammate.** **A537** — Use STAR. Situation: a developer disagreed with a defect I raised. Task: resolve without damaging the relationship. Action: reproduced the issue together, showed the SQL and sample keys, referenced the mapping rule, and agreed on the correct behavior. Result: developer fixed the code, and we added the case to regression tests. No blame, just evidence. **Q538 — Tell me about a time you missed a deadline.** **A538** — Situation: a test cycle was delayed because a dependency was late. Task: deliver what I could without compromising quality. Action: prioritized risk-based testing, communicated daily, escalated early with a revised plan. Result: critical paths were tested on time, non-critical deferred with sign-off. Lesson: escalate early, not at the deadline. **Q539 — Tell me about a time you took ownership beyond your role.** **A539** — Situation: a data quality issue was found in a report after my testing was signed off. Task: prevent recurrence. Action: traced the root cause to a join in the upstream pipeline, raised it with the upstream team, and proposed an automated check. Result: the check was added and the issue never recurred. **Q540 — Tell me about a time you mentored someone.** **A540** — Situation: a junior engineer was new to PySpark. Task: bring them up to speed. Action: pair-programmed on a validation job, reviewed their code line by line, and explained the "why" behind each check. Result: they independently built the next validation job with minimal review. **Q541 — Tell me about a time you learned a new tool quickly.** **A541** — Situation: a project required Databricks but I had only used Spark on-prem. Task: deliver within two weeks. Action: built a small end-to-end pipeline in a lab, read the docs, asked targeted questions, and validated against known-good SQL. Result: delivered the pipeline on time with full reconciliation. **Q542 — Tell me about a time you disagreed with your manager.** **A542** — Situation: manager wanted to skip regression testing to meet a deadline. Task: balance delivery and quality. Action: presented the risk of skipping (financial data, compliance), proposed a risk-based subset that covered critical paths. Result: manager agreed to the subset, we met the deadline, no defects escaped. **Q543 — Tell me about a time you failed.** **A543** — Situation: I missed a duplicate check in a pipeline and duplicates reached a Gold table. Task: fix and prevent. Action: found the cause (a non-idempotent write), fixed it, reran idempotently, and added a permanent uniqueness check. Result: no recurrence, and the check became standard across other pipelines. **Q544 — Why are you leaving your current role?** **A544** — [Be honest but positive.] I have grown a lot in my current role, and I am looking for a role that combines data engineering with a stronger validation and data-quality focus. This role fits that direction. **Q545 — Where do you see yourself in 5 years?** **A545** — Deepening expertise in data quality and platform reliability: leading validation strategy for a large data platform, mentoring junior engineers, and designing automated DQ frameworks. Not necessarily managing people, but owning a technical area end to end. **Q546 — What is your greatest strength?** **A546** — Proving data is right. I do not trust a green pipeline; I trust a reconciled one. I combine engineering knowledge with a testing mindset to find root cause and prevent recurrence. **Q547 — What is your greatest weakness?** **A547** — I dig deep for root cause, so I now timebox investigations and escalate with what I know. On the technical side, enterprise schedulers like Autosys and Control-M and shell scripting are lab-level for me and I am closing that gap. **Q548 — Why should we hire you?** **A548** — I bring three things: (1) hands-on ETL and validation experience in financial services and telecom, (2) automation skill in Python, SQL and PySpark, and (3) the right mindset — I look for the first hop where numbers diverge, prove it with data, and turn every defect into an automated check. **Q549 — How do you handle pressure?** **A549** — I prioritize by risk and impact, communicate early, and break the problem into small verifiable steps. Under pressure I lean on the rhythm: what I saw → where I narrowed it down → proof → fix → prevention. **Q550 — How do you handle ambiguity?** **A550** — I write down assumptions, ask targeted questions, and prototype small. If the requirement is unclear, I get the BA or business owner to confirm the rule before I build or test against it. --- ## SECTION 31 — AGILE / SCRUM / DELIVERY **Q551 — What is a sprint?** **A551** — A time-boxed iteration (usually 2 weeks) during which a set of user stories is completed and potentially shippable. **Q552 — What is a user story?** **A552** — A short description of a feature from the user's perspective: "As a [role], I want [feature] so that [benefit]." Acceptance criteria define when it is done. **Q553 — What is definition of done (DoD)?** **A553** — A shared checklist of what "done" means: code reviewed, unit tests pass, deployed to test, reconciliation passes, documentation updated, product owner accepts. For data stories, add "DQ checks pass" and "lineage updated". **Q554 — What is a retrospective?** **A554** — A meeting at the end of each sprint to reflect on what went well, what did not, and what to improve. Actions are tracked and reviewed next sprint. **Q555 — What is story point estimation?** **A555** — A relative measure of effort/complexity (Fibonacci: 1, 2, 3, 5, 8, 13). Not hours. Team velocity = average points completed per sprint. Used for forecasting. **Q556 — What is a burndown chart?** **A556** — A chart showing remaining work (story points or tasks) over the sprint. Ideally trends to zero by sprint end. A flat line early signals a problem. **Q557 — What is the difference between Scrum and Kanban?** **A557** — Scrum: time-boxed sprints, defined roles (PO, SM, team), ceremonies. Kanban: continuous flow, WIP limits, no fixed iterations. Both visualise work; Scrum is better for planned delivery, Kanban for support/ops. **Q558 — What is in-sprint testing?** **A558** — Testing performed during the sprint, not after. Tests are written and run as soon as the story is in the test environment. Definition of done includes passing tests. Reduces release risk. **Q559 — What is the offshore/onshore model at XXX?** **A559** — Client-facing onshore team handles requirements and coordination; offshore team (India) handles development and testing. Daily handover calls, shared tracker, and clear documentation bridge the gap. I have worked with global stakeholders in this model. **Q560 — How do you handle changing requirements mid-sprint?** **A560** — Assess impact, communicate to the PO, and either defer to next sprint (preferred) or swap out an equivalent story. Never silently expand scope. --- ## SECTION 32 — PRODUCTION OPERATIONS / ON-CALL **Q561 — What is on-call?** **A561** — A rotation where an engineer is available to respond to production incidents outside business hours. Responsibilities include triage, mitigation, communication, and postmortem. **Q562 — What is an incident severity level?** **A562** — S1/Critical: data loss, financial impact, major outage — page immediately. S2/High: degraded service, workaround exists — respond within hours. S3/Medium: minor impact — next business day. S4/Low: cosmetic — backlog. **Q563 — What is a runbook?** **A563** — A document describing how to detect, diagnose, mitigate, and escalate a specific incident. Includes symptoms, checks, commands, contacts, and rollback steps. Written before the incident, tested periodically. **Q564 — What is a postmortem?** **A564** — A blameless review after an incident: timeline, impact, root cause, what went well, what went wrong, and tracked preventive actions with owners and dates. Focus on system conditions, not individuals. **Q565 — What is a war room?** **A565** — A focused real-time incident response with all key people (engineering, ops, business, communications) in one channel/call. One incident commander coordinates; everyone else works on their piece. **Q566 — What is change management?** **A566** — A controlled process for deploying changes to production: request, review, approve, schedule, deploy, verify, rollback plan. Prevents unplanned changes from causing incidents. **Q567 — What is a rollback plan?** **A567** — A documented way to revert a deployment if it fails: previous artifact version, previous config, previous data state. Must be tested. For data pipelines, code rollback alone is not enough — you need data rollback/replay. **Q568 — What is a canary deployment?** **A568** — Deploy to a small subset of traffic/users first, monitor for errors, then roll out to everyone. For data pipelines: process a small date range or a sample first, verify, then run the full range. **Q569 — What is blue-green deployment?** **A569** — Two identical environments: blue (live) and green (new). Deploy to green, test, switch traffic from blue to green. Instant rollback by switching back. Not always applicable to data pipelines where state matters. **Q570 — What is a feature flag?** **A570** — A configuration toggle that enables/disables a feature without redeploying code. Useful for gradual rollout and quick disable if issues arise. For pipelines, use flags to switch between old and new logic. --- ## SECTION 33 — RESUME / CV SPECIFIC DEEP-DIVE QUESTIONS **Q571 — Walk me through your XXX project.** **A571** — [Use the prepared 20-second pitch.] Pipelines that ingest investment data from Azure Data Lake and S3, validate and reconcile it, and feed NAV, risk and fund-performance reporting. I owned the validation controls and tuned the SQL behind the KPIs. Key probes: sources and formats, layers, volumes, DQ controls, how failures were reported, compliance-sensitive meaning, Jenkins deployment, SQL Server changes, biggest defect found. **Q572 — Walk me through your telecom churn project.** **A572** — [Prepared pitch.] PySpark ETL on usage, demographic and log data, SQL churn features, ETL testing across lake and warehouse, SSIS batches, churn model and Power BI dashboards. Key probes: input formats, churn features built, window functions used (LAG for usage trend, ROW_NUMBER for latest record), SSIS package structure, ETL tests, model type and metric, Power BI KPIs, dev-test-prod promotion. **Q573 — Walk me through your XXX project.** **A573** — [Prepared pitch.] Transaction analytics in SQL and Python, PySpark ETL from S3 and Blob, reconciliation and pipeline-quality checks, Scikit-learn behavior model, Power BI dashboards. Key probes: purchase frequency, AOV, conversion, cohort retention, segments, model and metric, PySpark specifics, DAX measures, anomaly monitoring. **Q574 — What was your biggest technical challenge at XXX?** **A574** — [Use Story A or E from Section 2.] A reconciliation mismatch after a release where the cause was a join fan-out on an effective-dated reference table. I proved it by tracing one security end to end, fixed the join grain, reran idempotently, and added two permanent checks. **Q575 — What did you build in your RHEL lab?** **A575** — Hadoop HDFS/YARN, Hive Metastore, Iceberg, Kafka (KRaft), Flink, Airflow, Trino, Prometheus/Grafana. I set up a small end-to-end pipeline to understand how these components fit together and how to test them. **Q576 — Why did you design the two AWS reference architectures?** **A576** — To demonstrate production controls — idempotency, quarantine, DQ gates, replay, multi-region DR — that I would apply in a real platform. I tested parts in my lab. I would not claim I operated every AWS component in production. **Q577 — What is your experience with Snowflake/BigQuery/GCP?** **A577** — [Use gap template.] I haven't used it in a project. The validation approach is the same: counts, aggregates, row-level diff, business rules. In Snowflake I would use TIME TRAVEL, QUALIFY, HASH functions; in BigQuery, partitions and FARM_FINGERPRINT. I have done the same in SQL Server and Databricks, and I ramp up quickly. **Q578 — What is your experience with Informatica/Ab Initio/DataStage?** **A578** — [Use gap template.] I haven't built in those tools. Testing is the same: counts, transformation logic, rejects, restartability and logs. I would first learn the tool's reject mechanism and monitoring, then map each rule to the checks I already automate. **Q579 — How do you stay current with data engineering trends?** **A579** — I follow AWS/Azure/Databricks release notes, read engineering blogs, practice in my lab, and take relevant certifications. I also learn from production incidents — every defect teaches something new about the platform. **Q580 — What certifications do you hold?** **A580** — IBM Data Analyst Professional Certificate (Coursera), Data Science & Analytics Specialist (ITVEDANT), Hardware & Networking Technician (Silicon Valley Institute). [Add any new certs if earned.] --- ## SECTION 34 — DOMAIN-SPECIFIC BANKING QUESTIONS (XXX) **Q581 — What is fund accounting?** **A581** — The process of recording and reporting a fund's assets, liabilities, income, and expenses. Produces NAV per unit, which is the price at which investors buy/sell units. Compliance-sensitive because it determines investor payouts. **Q582 — What is NAV calculation?** **A582** — NAV = (Total Assets − Total Liabilities) / Units Outstanding. Assets include securities at market value (quantity × price × FX rate) plus cash and receivables. Liabilities include payables, accruals and expenses. A single mispriced security or wrong FX rate affects every investor. **Q583 — What are NAV validation checks?** **A583** — Position roll-forward (opening + buys − sells ± corporate actions = closing), market value = qty × price × FX, portfolio weights sum to 100%, day-over-day NAV change within expected band, no missing prices or FX rates, corporate actions applied correctly, no duplicate positions. **Q584 — What is fund performance?** **A584** — Return of the fund vs a benchmark over a period, usually expressed as a percentage. Time-weighted return (TWR) removes the effect of cash flows; money-weighted return (MWR/IRR) reflects the investor's actual return. Both are validated against custodian and index data. **Q585 — What is risk reporting?** **A585** — Measures such as volatility (standard deviation), beta (sensitivity to market), Sharpe ratio (return per unit of risk), VaR (Value at Risk), tracking error (deviation from benchmark), and drawdown. Data must be accurate and reconciled. **Q586 — What is a fund KPI?** **A586** — Key Performance Indicator — a metric the business uses to monitor fund health: AUM, net flows, expense ratio, return vs benchmark, tracking error, Sharpe ratio. KPIs must be defined, documented, and validated against source. **Q587 — What is a corporate action and how does it affect a portfolio?** **A587** — A corporate action is an event initiated by a company that affects its securities: dividends (cash or stock), splits (share count changes, price adjusts), mergers (one security replaced by another), rights issues, spin-offs. Each must be applied to both quantity and price correctly, and the effect must be reflected in NAV and performance. **Q588 — What is a trade lifecycle?** **A588** — Order → execution → confirmation → clearing → settlement. US equities settle T+1 since 28 May 2024. Settlement date must be a business day; a failed settlement is an operational risk. Data validation must confirm trade date, settle date, and status transitions. **Q589 — What is the difference between trade date and settlement date?** **A589** — Trade date: when the transaction is agreed. Settlement date: when cash and securities actually exchange. Accounting may use trade date or settlement date depending on the standard (trade-date accounting is common for funds). **Q590 — What is custodian reconciliation?** **A590** — Comparing the fund's internal position and cash records against the custodian bank's records. Differences must be investigated — they could indicate a missing trade, a wrong price, or a corporate action not applied. **Q591 — What is an ISIN?** **A591** — International Securities Identification Number: 12 characters — 2-letter country code, 9-character alphanumeric national identifier, 1 check digit. Used globally to identify securities. **Q592 — What is a CUSIP / SEDOL / ticker?** **A592** — CUSIP: 9-character North American identifier. SEDOL: 7-character UK/international identifier. Ticker: exchange-specific short symbol. All must be mapped to the same security in the security master. **Q593 — What is a security master?** **A593** — The golden record of all securities: identifiers (ISIN, CUSIP, SEDOL, ticker), name, type, currency, exchange, effective dates, status. Reference data quality directly affects every downstream calculation. **Q594 — What is an FX rate and how is it validated?** **A594** — The rate to convert one currency to another. Validated by: source (vendor), timestamp/date (spot vs forward), inverse consistency (if USD/EUR = 1.1, EUR/USD ≈ 0.909), and cross-rate consistency. A wrong FX rate misprices every foreign-currency position. **Q595 — What is a trading calendar and why validate against it?** **A595** — A list of trading days and holidays per exchange. Used to decide which days should have data. A missing trading day with no prices is a defect; a non-trading day with prices is also a defect. Half-days (early close) need special handling. **Q596 — What is PII in banking data?** **A596** — Personally Identifiable Information: name, address, SSN, account number, email, phone. Must be masked in non-prod, encrypted at rest, and access-audited. GDPR and other regulations impose retention and residency limits. **Q597 — What is KYC and AML?** **A597** — Know Your Customer and Anti-Money Laundering: regulatory requirements to verify customer identity and monitor for suspicious activity. Data pipelines supporting KYC/AML must have audit trails and strict access controls. **Q598 — What is MiFID / EMIR / Dodd-Frank?** **A598** — Regulatory frameworks for financial markets: MiFID (EU, markets in financial instruments), EMIR (EU, derivatives reporting), Dodd-Frank (US, post-2008 reform). Each imposes reporting, retention, and data quality requirements. **Q599 — What is a benchmark index?** **A599** — A reference portfolio (e.g., S&P 500) used to measure fund performance. Index constituents, weights, and returns must be sourced and reconciled. Changes to the index (rebalancing) must be applied on the effective date. **Q600 — What is performance attribution?** **A600** — Decomposing a fund's return vs benchmark into sources: allocation (sector/region weights) and selection (security picks). Requires accurate position, price, and FX data at each period. A single error propagates through the attribution. --- ## SECTION 35 — MISC / HIGH-VALUE ADDITIONS **Q601 — What is the difference between a data engineer and a data analyst?** **A601** — Data engineer builds and maintains pipelines, storage, and infrastructure — focuses on reliability, scale, and data quality. Data analyst queries and interprets data for business insights — focuses on visualization, statistics, and storytelling. The roles overlap at SQL and data modeling. **Q602 — What is a data platform?** **A602** — The combination of storage, compute, orchestration, catalog, governance, and monitoring that ingests, transforms, serves, and governs data. A platform is more than a pipeline — it includes the controls, standards, and self-service capabilities for many teams. **Q603 — What is a data product?** **A603** — A curated, governed dataset with a defined owner, SLA, schema, quality contract, and consumers. Treated as a product, not a byproduct of a pipeline. Gold tables in a well-run platform are data products. **Q604 — What is a data mesh?** **A604** — A decentralized approach where domain teams own their data products, publish them with contracts, and a central platform provides self-service infrastructure and governance. Contrasts with a central data team owning everything. **Q605 — What is a data fabric?** **A605** — A metadata-driven approach that connects disparate data sources with automated integration, governance, and access. Emphasizes virtualization and active metadata over physical consolidation. **Q606 — What is the difference between a data lake and a data swamp?** **A606** — A data lake is organized, cataloged, governed storage. A data swamp is a data lake without governance — no catalog, no schema, no quality, no ownership. The difference is discipline, not technology. **Q607 — What is a data contract in practice?** **A607** — A versioned agreement between producer and consumer: schema, semantics, keys, nullability, freshness, compatibility rules, ownership, and SLA. Enforced in CI/CD (schema gate) and monitored in production (freshness, schema drift). **Q608 — What is a data SLA vs SLO vs SLI?** **A608** — SLI (Service Level Indicator): a measured metric (freshness, completeness, latency). SLO (Objective): the target for the SLI (freshness < 5 minutes 99% of the time). SLA (Agreement): a commitment with consequences if the SLO is missed. **Q609 — What is data lineage and why does it matter?** **A609** — Lineage traces a data element from source through transformations to target and report. Matters for impact analysis (what breaks if I change this?), root cause (where did this wrong number come from?), and compliance (prove data provenance). **Q610 — What is a data catalog?** **A610** — A metadata repository describing datasets, schemas, locations, owners, lineage, and quality. Enables discovery, governance, and self-service. Examples: Glue Data Catalog, Unity Catalog, Purview, Datahub. **Q611 — What is a data steward?** **A611** — The person accountable for a data domain's quality, definitions, and governance. Not necessarily technical — often a business role. Approves definitions and resolves conflicts. **Q612 — What is a data owner?** **A612** — The person or team accountable for a dataset's availability, quality, security, and lifecycle. Approves access and signs off on changes. **Q613 — What is the difference between a data owner and a data steward?** **A613** — Owner: accountable for the dataset (access, lifecycle, SLA). Steward: responsible for the day-to-day quality, definitions, and metadata. Owner decides; steward maintains. **Q614 — What is a data quality scorecard?** **A614** — A dashboard showing DQ metrics per dataset: completeness %, validity %, uniqueness %, freshness, reconciliation pass rate, defect count. Used to track trends and prioritize remediation. **Q615 — What is a data quality rule?** **A615** — A testable condition on data: not-null, unique, in-range, in-domain, referential integrity, transformation rule, cross-field consistency. Each rule has a severity (critical/warning) and a threshold. **Q616 — What is data profiling?** **A616** — Statistical analysis of data to understand its structure and content: null %, distinct count, min/max, distribution, format patterns, outliers. Done before building a pipeline or a DQ rule to set realistic expectations. **Q617 — What is data cleansing?** **A617** — Fixing data quality issues: trimming, case normalization, date parsing, standardizing codes, deduplicating, correcting known errors. Cleansing rules must be documented in the mapping and tested. **Q618 — What is data enrichment?** **A618** — Adding context to data from other sources: joining a security master to add sector, joining FX rates to convert currency, adding a derived band (age band, income band). Enrichment changes the grain if not careful — validate row counts. **Q619 — What is data masking vs tokenization vs encryption?** **A619** — Masking: replace part of a value (e.g., ****1234). Tokenization: replace with a token that maps back via a secure vault. Encryption: reversible transformation with a key. Choose based on whether the original must be recoverable and who can reverse it. **Q620 — What is a data retention policy?** **A620** — Rules for how long data is kept and when it is deleted. Driven by regulation (7 years for financial), business need, and cost. Implemented via lifecycle policies (S3), TTL (Redis), and purge jobs. **Q621 — What is the difference between hot, warm, and cold data?** **A621** — Hot: frequently accessed, low latency (Redis, Aurora). Warm: occasional access, moderate latency (S3 Standard, Redshift). Cold: rare access, high latency but cheap (S3 Glacier, tape). Tier data based on access patterns. **Q622 — What is a data lakehouse?** **A622** — Combines the cheap storage of a data lake with the ACID transactions, schema, and performance of a warehouse, using table formats like Delta Lake or Iceberg. Enables one platform for BI and ML. **Q623 — What is a semantic layer?** **A623** — A governed set of business definitions, measures, dimensions, and relationships shared across reports and tools. Examples: Power BI dataset, dbt metrics, Looker LookML. Ensures one number for one metric. **Q624 — What is a metric store?** **A624** — A centralized repository of business metrics with definitions, owners, and lineage. Enables consistent reporting across tools. Prevents "every team has its own AUM definition". **Q625 — What is data governance?** **A625** — The policies, standards, roles, and processes that ensure data is accurate, available, secure, and compliant. Includes catalog, lineage, quality, access, retention, and stewardship. **Q626 — What is data stewardship vs data governance?** **A626** — Governance: the framework (policies, standards, roles). Stewardship: the execution (day-to-day quality, metadata, issue resolution). Governance sets the rules; stewardship follows them. **Q627 — What is a data quality incident?** **A627** — An event where data fails to meet its contract: missing, late, wrong, or non-compliant. Treated like any incident: detect, triage, mitigate, communicate, postmortem, prevent. **Q628 — What is a data SLO example?** **A628** — "Gold table freshness < 5 minutes 99% of trading days." The SLI is the observed freshness, the SLO is the 99% target, and the SLA is what the business is promised (with consequences). **Q629 — What is the difference between a data defect and a code defect?** **A629** — Code defect: the logic is wrong (e.g., wrong join, wrong formula). Data defect: the input is wrong (e.g., source sent bad data, late file, missing FX rate). Both manifest as wrong output but require different fixes. **Q630 — What is a data quality firewall?** **A630** — A gate that blocks bad data from reaching consumers. Includes schema validation, DQ rules, reconciliation, and quarantine. Critical rules block publication; warnings allow publication with visibility. --- ## FINAL COUNT **Original bank:** 445 Q&As (Sections 1–21) **New additions:** 185 Q&As (Sections 22–35, Q446–Q630) **Total:** 630 Q&As covering every domain in the uploaded files plus gaps. --- ## WHAT WAS STILL MISSING — SUMMARY | Gap Area | Why It Matters | New Section | |---|---|---| | SSIS deep-dive | On your CV — must defend | 22 | | Databricks deep-dive | On your CV — must defend | 23 | | Power BI / DAX deep-dive | On your CV — must defend | 24 | | Git / Jenkins / Docker / Terraform | On your CV — must defend | 25 | | Linux / shell | JD asks — CV gap | 26 | | Advanced SQL (recursive CTE, PIVOT, MERGE, isolation, deadlock, index) | Common senior questions | 27 | | Data modeling (Kimball/Inmon, Data Vault, bridge, junk, mini-dim, snapshot) | Common DWH questions | 28 | | Testing methodology (verification/validation, traceability, entry/exit, defect metrics) | QA-focused role | 29 | | Behavioral / STAR | Every interview | 30 | | Agile / Scrum / delivery | XXX context | 31 | | Production operations / on-call | Senior-level expectation | 32 | | CV-specific deep-dive | Interviewer goes 2–3 levels deep | 33 | | Banking domain (NAV, fund accounting, corporate actions, ISIN, regulation) | XXX project | 34 | | Misc (data mesh, fabric, mesh, catalog, steward, retention, hot/warm/cold) | Senior-level breadth | 35 | --- **Total Q&A bank now: 630 questions.** This covers every line on your CV, every domain in both architecture diagrams, every failure area, and every common interview pattern for ETL/Data Validation/Data Engineer roles at XXX, TCS, Accenture, Amazon, Microsoft, and product firms. --- Source file: Pasted text (3)(2).txt # Still Missing — Additional Q&A Bank (Q631–Q800) After reviewing the 630-question bank against the combined uploaded files and real interview patterns, the following **170 questions are still missing**. These are the highest-value gaps for ETL/Data Engineer roles at XXX, TCS, Accenture, Amazon, Microsoft, and product firms. --- ## SECTION 36 — SNOWFLAKE DEEP DIVE (JD MENTIONS — YOUR CV IS THIN) **Q631 — What is Snowflake's architecture?** **A631** — Three layers: (1) Storage layer — compressed, columnar micro-partitions on cloud object storage (S3/Azure Blob/GCS). (2) Compute layer — independent virtual warehouses (clusters) that query storage. (3) Cloud services layer — metadata, optimizer, security, transactions. Compute and storage scale independently, and multiple warehouses can query the same data without contention. **Q632 — What is a micro-partition?** **A632** — Snowflake automatically partitions tables into contiguous, immutable micro-partitions of 50–500 MB, stored columnar and compressed. Each micro-partition stores metadata (min/max per column) used for pruning. Users do not manage partitions — Snowflake does. **Q633 — What is a virtual warehouse?** **A633** — A cluster of compute resources that executes queries. Sizes range from X-Small to 6X-Large. Multi-cluster warehouses auto-scale concurrency. Warehouses can be started/stopped/resized independently and billed per second (60-second minimum). **Q634 — What is Time Travel in Snowflake?** **A634** — Query data as of a past point: `SELECT * FROM t AT (TIMESTAMP => '2026-10-01 00:00:00'::timestamp)` or `AT (OFFSET => -3600)` or `BEFORE (STATEMENT => 'query_id')`. Default retention 1 day (Standard) or up to 90 days (Enterprise). Used to compare before/after a load or recover from a bad write. **Q635 — What is Fail-safe?** **A635** — A 7-day period after Time Travel expires during which Snowflake can recover data, but only by Snowflake support. Not user-accessible. Not a substitute for backups. **Q636 — What is a zero-copy clone?** **A636** — `CREATE TABLE clone CLONE original` creates a new table that shares the same micro-partitions initially and diverges as data changes. No data is copied. Used for test environments and validation without storage cost. **Q637 — What is a clustering key?** **A637** — A column (or expression) defined on a table that co-locates related rows in the same micro-partitions, improving pruning for large tables. `ALTER TABLE t CLUSTER BY (date, region)`. Automatic clustering maintains it as data changes (costs credits). **Q638 — What is a materialized view?** **A638** — A pre-computed result stored and automatically maintained by Snowflake when base tables change. Improves performance for repeated expensive queries. Costs storage and compute for maintenance. Not the same as a regular view (which is just a stored query). **Q639 — What is a secure view?** **A639** — A view whose definition and underlying data are hidden from users who don't have access. Prevents users from seeing the SQL logic or querying base tables directly. Used for row/column-level security. **Q640 — What is a resource monitor?** **A640** — A credit quota on a warehouse or account. Alerts, suspends, or notifies when usage hits thresholds. Essential for cost control. **Q641 — What is query acceleration service?** **A641** — Snowflake offloads portions of a query to shared compute resources, speeding up scans on large tables with selective filters. Enabled per warehouse. **Q642 — What is search optimization service?** **A642** — An index-like service that speeds up point-lookup and substring searches on large tables. Enabled per table. Costs additional storage and compute. **Q643 — What is a COPY INTO with VALIDATION_MODE?** **A643** — Loads data and returns errors without loading: `COPY INTO t FROM @stage FILE_FORMAT=(...) VALIDATION_MODE='RETURN_ERRORS'`. Use `VALIDATE(t, JOB_ID)` to see rejected rows. Essential for testing file loads before committing. **Q644 — What is the difference between a stage and a storage integration?** **A644** — A stage is a named location in Snowflake (internal or external) pointing to files. A storage integration is a Snowflake object that stores the IAM role/trust relationship for accessing external cloud storage, avoiding embedding credentials in DDL. **Q645 — What is a stream and a task in Snowflake?** **A645** — A stream captures CDC changes (insert/update/delete) on a table. A task runs SQL on a schedule or when a stream has data. Together they implement incremental pipelines natively in Snowflake without external orchestration. **Q646 — What is the difference between a transient and a permanent table?** **A646** — Permanent: full Time Travel and Fail-safe. Transient: no Fail-safe, limited Time Travel — cheaper, for staging/intermediate data. Temporary: session-scoped, auto-dropped. **Q647 — What is Snowpipe?** **A647** — Snowflake's continuous ingestion service. Files land in a stage, Snowpipe detects and loads them automatically (via event notification or REST API). Serverless or warehouse-based. Used for near-real-time file ingestion. **Q648 — How do you validate Snowflake loads?** **A648** — (1) `COPY_HISTORY` for load status. (2) `VALIDATE` for rejected rows. (3) Compare row counts source vs target. (4) Use Time Travel to compare before/after. (5) Check `STL_LOAD_ERRORS` or `VALIDATE` output for parse errors. (6) Query `ACCOUNT_USAGE.LOAD_HISTORY` for history. **Q649 — What is Snowflake's enforcement of constraints?** **A649** — Snowflake enforces NOT NULL but **does not enforce** PRIMARY KEY, FOREIGN KEY, or UNIQUE. They are informational for the optimizer. Always validate uniqueness and referential integrity in the pipeline. **Q650 — How do you do row-level security in Snowflake?** **A650** — Row access policies: `CREATE ROW ACCESS POLICY ... AS (col) RETURNS BOOLEAN -> ...` then `ALTER TABLE ... ADD ROW ACCESS POLICY ... ON (col)`. Applied at query time. Test with different roles. **Q651 — How do you do column-level security in Snowflake?** **A651** — Masking policies: `CREATE MASKING POLICY ... AS (val STRING) RETURNS STRING -> CASE WHEN current_role() IN (...) THEN val ELSE '***' END`. Apply with `ALTER TABLE ... MODIFY COLUMN ... SET MASKING POLICY ...`. Also tag-based masking. **Q652 — What is a Snowflake share?** **A652** — A secure, governed way to share data with another Snowflake account (or reader account) without copying. The provider defines a share with specific tables/views; the consumer mounts it as a database. Used for data exchange and marketplace. --- ## SECTION 37 — BIGQUERY DEEP DIVE (JD MENTIONS — YOUR CV IS THIN) **Q653 — What is BigQuery's architecture?** **A653** — Serverless, multi-tenant. Storage is Colossus (Google's distributed file system) with Capacitor columnar format. Compute is Dremel (tree architecture with mixers and leaf nodes). Separates storage and compute. Query is billed by bytes scanned (on-demand) or slots (flat-rate). **Q654 — What is a partitioned table in BigQuery?** **A654** — A table split by a date/timestamp column or ingestion time into partitions. Queries with a partition filter scan only relevant partitions — reduces cost and improves speed. `CREATE TABLE ... PARTITION BY DATE(col)`. **Q655 — What is a clustered table in BigQuery?** **A655** — Rows within partitions are sorted by up to 4 columns. Improves filter and aggregate performance on those columns. `CLUSTER BY col1, col2`. Combine with partitioning for best results. **Q656 — What is the difference between Legacy SQL and Standard SQL?** **A656** — Legacy SQL is older, uses `[project:dataset.table]`, has different functions. Standard SQL is ANSI-compliant, supports `QUALIFY`, `ARRAY`, `STRUCT`, standard joins. Always use Standard SQL. **Q657 — What is a slot in BigQuery?** **A657** — A unit of computational capacity. On-demand queries use a shared pool (up to 2,000 slots per project). Flat-rate reservations buy dedicated slots. Slots determine concurrency and speed. **Q658 — What is the query cost model?** **A658** — On-demand: $6.25 per TB scanned (US). Flat-rate: monthly commitment for slots. Minimize scan by: selecting only needed columns, partitioning, clustering, using `LIMIT` carefully (does not reduce scan), and materializing intermediate results. **Q659 — What is a materialized view in BigQuery?** **A659** — A pre-computed view automatically refreshed from the base table. Improves performance for repeated queries. BigQuery smart-tuning can also use materialized views automatically for matching queries. **Q660 — What is the difference between a table, a view, and a materialized view?** **A660** — Table: stored data. View: stored query, executed at query time. Materialized view: stored result, automatically refreshed. Choose by freshness vs performance trade-off. **Q661 — What is a wildcard table?** **A661** — Query multiple tables matching a pattern: `SELECT * FROM \`project.dataset.table_*\``. Useful for date-sharded tables. Use `_TABLE_SUFFIX` to filter. Cost depends on tables scanned — filter tightly. **Q662 — What is a nested/repeated field in BigQuery?** **A662** — BigQuery supports `STRUCT` (nested) and `ARRAY` (repeated) types. Query with `UNNEST(array)` to flatten. Denormalized but efficient — avoids joins. Common for event data. **Q663 — How do you validate BigQuery loads?** **A663** — (1) `INFORMATION_SCHEMA.JOBS` for job history and bytes billed. (2) `INFORMATION_SCHEMA.TABLES` for row counts. (3) Partition filter to compare specific dates. (4) `EXCEPT DISTINCT` for row-level comparison. (5) `FARM_FINGERPRINT` for row hashing. (6) Check `_PARTITIONTIME` or `_PARTITIONDATE` for ingestion. **Q664 — What is the difference between `EXCEPT` and `EXCEPT DISTINCT` in BigQuery?** **A664** — BigQuery's `EXCEPT DISTINCT` is set-based (removes duplicates). There is no `EXCEPT ALL` in BigQuery — use `FULL OUTER JOIN` or group-by counts for multiset comparison. **Q665 — What is the difference between `QUALIFY` and `WHERE` with a subquery?** **A665** — `QUALIFY` filters on window function results directly: `SELECT ..., ROW_NUMBER() OVER (...) rn FROM t QUALIFY rn = 1`. Cleaner than a subquery. Supported in BigQuery, Snowflake, Databricks SQL, Teradata. **Q666 — What is BigQuery BI Engine?** **A666** — An in-memory analysis service that accelerates BI queries (Looker Studio, Looker, Tableau). Caches frequently accessed data. Reduces query cost and latency. **Q667 — What is BigQuery Omni?** **A667** — Runs BigQuery queries on data in AWS S3 or Azure Blob without moving it. Uses Anthos. Useful for multi-cloud without data duplication. **Q668 — What is a BigQuery reservation?** **A668** — A flat-rate commitment for slots. Assign to projects/folders. Includes autoscaling (buy more slots on demand within limits). Better for predictable, high-volume workloads. --- ## SECTION 38 — REDSHIFT DEEP DIVE (JD MENTIONS — YOUR CV SAYS EXPOSURE) **Q669 — What is Redshift's architecture?** **A669** — A cluster with a leader node (query planning, coordination) and compute nodes (storage + execution). Columnar storage, massively parallel processing (MPP). RA3 nodes separate compute and storage (managed storage on S3). DC2 nodes are compute+storage. **Q670 — What is a distribution key?** **A670** — Determines how rows are distributed across nodes. Options: EVEN (round-robin), ALL (copied to every node — for small dimensions), KEY (hash on a column — for joins). Choose based on join patterns. Wrong choice causes data skew. **Q671 — What is a sort key?** **A671** — Determines the order of rows on disk within each node. Compound sort key (ordered) or interleaved sort key (equal weight). Improves range filter and join performance via zone maps (min/max per block). **Q672 — What is a zone map?** **A672** — Redshift stores min/max values per 1 MB block. Queries with range filters skip blocks that don't match — like partition pruning at the block level. Effective when data is sorted on the filter column. **Q673 — What is VACUUM in Redshift?** **A673** — Reclaims space from deleted rows and re-sorts rows according to the sort key. Run after large deletes/updates. `VACUUM FULL` for full re-sort, `VACUUM SORT ONLY` for sort, `VACUUM DELETE ONLY` for space. Auto-vacuum exists but may lag. **Q674 — What is ANALYZE in Redshift?** **A674** — Updates table statistics used by the query planner. Stale stats cause bad plans. `ANALYZE table`. Auto-analyze exists but can lag. Run after large loads. **Q675 — What is a WLM queue?** **A675** — Workload Management — configures how queries are queued and resources allocated. Automatic WLM (recommended) or manual. Controls concurrency and memory per queue. Query priority and user groups. **Q676 — What is Redshift Spectrum?** **A676** — Queries data directly in S3 (Parquet, ORC, JSON) without loading into Redshift. Uses an external schema and external tables. Good for infrequent access to large cold data. **Q677 — What is AQUA (Advanced Query Accelerator)?** **A677** — Hardware-accelerated cache for RA3 clusters that pushes computation closer to storage. Speeds up scans and aggregations. No code changes needed. **Q678 — What is a materialized view in Redshift?** **A678** — A pre-computed result refreshed manually or on a schedule. `REFRESH MATERIALIZED VIEW`. Can be auto-refreshed. Improves performance for repeated complex queries. **Q679 — How do you validate Redshift loads?** **A679** — (1) `STL_LOAD_ERRORS` for COPY failures. (2) `SYS_LOAD_ERROR_DETAIL` on Serverless. (3) `STL_LOAD_COMMITS` for load history. (4) Row counts and sums by slice. (5) Check distribution skew: `SELECT slice, COUNT(*) FROM table GROUP BY slice`. (6) `SVV_TABLE_INFO` for skew and unsorted rows. **Q680 — What is the difference between COPY and INSERT in Redshift?** **A680** — COPY is massively parallel, loads from S3/DynamoDB/EMR in bulk, recommended for all bulk loads. INSERT is row-by-row, slow, for small sets. COPY supports compression, columnar formats, and error handling. **Q681 — What is the difference between a leader node and a compute node?** **A681** — Leader: receives queries, parses, plans, coordinates. Compute: stores data and executes. Client never connects directly to compute nodes. Concurrency scaling adds transient clusters for read-heavy workloads. **Q682 — What is concurrency scaling in Redshift?** **A682** — Automatically adds transient clusters when query queues are full. Bills per second. Handles spikes without provisioning for peak. Only for read queries. --- ## SECTION 39 — dbt (DATA BUILD TOOL) — NOT ON YOUR CV BUT COMMON **Q683 — What is dbt?** **A683** — Data build tool — a transformation framework that lets analysts and engineers write SQL SELECT statements, and dbt handles materialization (table/view/incremental), dependency management, testing, documentation, and lineage. Runs on Snowflake, BigQuery, Redshift, Databricks, Postgres. **Q684 — What is a dbt model?** **A684** — A SQL file with a SELECT statement that defines a table or view. dbt compiles it, resolves refs, and materializes it in the warehouse. Example: `models/stg_trades.sql`. **Q685 — What is a dbt source?** **A685** — A declaration of a raw table in the warehouse (YAML) with freshness checks, tests, and documentation. Models reference sources with `{{ source('raw', 'trades') }}`. **Q686 — What is a dbt ref?** **A686** — `{{ ref('model_name') }}` — references another model, letting dbt build the dependency graph (DAG) automatically. Ensures correct build order. **Q687 — What materializations does dbt support?** **A687** — view (no storage), table (full rebuild), incremental (append/merge new rows), ephemeral (CTE-like, not stored), materialized view (warehouse-specific). Choose by data size and freshness. **Q688 — What is dbt testing?** **A688** — (1) Schema tests: not_null, unique, accepted_values, relationships — declared in YAML. (2) Custom tests: SQL queries that return failing rows. (3) `dbt test` runs all tests. (4) Tests can be severity warn or error. (5) Freshness tests on sources. **Q689 — What is a dbt macro?** **A689** — A reusable Jinja function. Example: `{% macro cents_to_dollars(column) %} ({{ column }} / 100.0) {% endmacro %}`. Used to DRY up SQL and abstract warehouse-specific syntax. **Q690 — What is dbt documentation?** **A690** — YAML descriptions for models, columns, sources. `dbt docs generate` builds a lineage graph and searchable docs. `dbt docs serve` hosts them locally. **Q691 — What is dbt incremental?** **A691** — A materialization that appends/merges only new/changed rows. Uses `{{ this }}` to reference the existing table and a filter (e.g., `WHERE updated_at > (SELECT MAX(updated_at) FROM {{ this }})`). Requires a unique key and a strategy (append, merge, delete+insert). **Q692 — What is dbt snapshot?** **A692** — Implements SCD Type 2 in dbt: captures changes to a source table over time. Configured in a YAML with a unique key, updated_at, and strategy (timestamp or check). dbt manages the valid_from/valid_to and dbt_valid_to columns. **Q693 — What is dbt Cloud vs dbt Core?** **A693** — dbt Core: open-source CLI. dbt Cloud: hosted service with scheduler, IDE, CI/CD, docs hosting, and environment management. Choose Core for flexibility, Cloud for managed orchestration. --- ## SECTION 40 — DISTRIBUTED SYSTEMS CONCEPTS (SENIOR QUESTIONS) **Q694 — What is CAP theorem?** **A694** — In a distributed system, you can guarantee at most two of: Consistency (all nodes see the same data), Availability (every request gets a response), Partition tolerance (system works despite network partitions). Since partitions happen, real systems choose CP or AP. Examples: HBase CP, Cassandra AP, Kafka AP with tunable consistency. **Q695 — What is eventual consistency?** **A695** — After a write, replicas may temporarily diverge but eventually converge. Reads may return stale data. Acceptable for many use cases (social feeds, analytics) but not for financial transactions. Balance with read-your-writes or strong consistency where needed. **Q696 — What is the two-phase commit (2PC)?** **A696** — A distributed transaction protocol: phase 1 (prepare) asks all participants to vote; phase 2 (commit/abort) acts on the vote. Blocking if the coordinator fails. Used in databases but avoided in high-scale systems due to blocking and latency. **Q697 — What is the saga pattern?** **A697** — A long-running distributed transaction split into local transactions, each with a compensating action for rollback. Choreography (events) or orchestration (coordinator). Used in microservices and data pipelines for multi-step operations that must be reversible. **Q698 — What is idempotency in distributed systems?** **A698** — An operation that produces the same result whether executed once or many times. Essential because networks and processes fail, and retries are inevitable. Achieved with idempotency keys, conditional writes, or upserts. **Q699 — What is consistent hashing?** **A699** — A technique for distributing data across nodes such that adding/removing a node remaps only a fraction of keys. Used in distributed caches, sharded databases, and Kafka partitioning. Avoids full rehashing on scaling. **Q700 — What is a leader election?** **A700** — A protocol by which distributed nodes agree on a single leader to coordinate writes. Examples: Raft, Paxos, ZooKeeper, KRaft. Used in Kafka controllers, database replicas, and distributed locks. **Q701 — What is a distributed lock?** **A701** — A mechanism to ensure only one process performs an action at a time across nodes. Implemented with Redis (SET NX EX), ZooKeeper, or a database row lock. Must handle expiry, fencing tokens, and failure of the lock holder. **Q702 — What is the outbox pattern?** **A702** — Write to the database and an outbox table in the same transaction; a separate process reads the outbox and publishes events. Ensures events are not lost if the message broker is down. Common in event-driven systems. **Q703 — What is CQRS?** **A703** — Command Query Responsibility Segregation — separate models for writes (commands) and reads (queries). Allows optimizing each independently. Often paired with event sourcing. Adds complexity but scales well. **Q704 — What is event sourcing?** **A704** — Storing all changes as a sequence of immutable events rather than the current state. State is derived by replaying events. Enables audit, time travel, and rebuilding projections. Used in financial systems and audit-heavy domains. **Q705 — What is the difference between at-least-once and exactly-once?** **A705** — At-least-once: a message may be delivered multiple times (requires idempotent processing). Exactly-once: each message processed once — hard to guarantee end-to-end, requires transactional coordination between source, processing, and sink. Kafka supports exactly-once within Kafka; end-to-end depends on the sink. --- ## SECTION 41 — PERFORMANCE TUNING DEEP DIVE **Q706 — How do you tune a slow SQL query?** **A706** — (1) Read the actual execution plan. (2) Check scans vs seeks — add index if scanning. (3) Look for key lookups — add INCLUDE columns. (4) Find implicit conversions — align data types. (5) Check for non-sargable predicates — rewrite with ranges. (6) Update statistics. (7) Consider a covering index. (8) Rewrite correlated subqueries as joins or window functions. (9) Check for parameter sniffing. (10) Measure before/after with logical reads. **Q707 — What is parameter sniffing?** **A707** — SQL Server compiles a plan for the first parameter values it sees, which may be optimal for those but bad for others. Symptoms: a query fast for one parameter, slow for another. Fixes: OPTION (RECOMPILE), OPTIMIZE FOR, or parameterized queries with plan guides. **Q708 — What is index fragmentation and how do you fix it?** **A708** — Page splits from inserts/updates cause index pages to become fragmented. Reorganize (online, light) or rebuild (offline, full). Check `sys.dm_db_index_physical_stats`. Fragmentation > 30% rebuild, 10–30% reorganize. **Q709 — What is the difference between a hash join, merge join, and nested loop join?** **A709** — Nested loop: for each row in the outer, scan the inner (good for small sets or indexed inner). Hash: build a hash table on the smaller side, probe with the larger (good for large unsorted sets). Merge: both sides sorted, merge (good when both sides have an index on the join key). The optimizer chooses based on statistics. **Q710 — How do you tune a slow Spark job?** **A710** — (1) Read Spark UI: longest stage, shuffle, spill, skew. (2) Filter/project early. (3) Broadcast small dimensions. (4) Handle skew with salting or AQE. (5) Tune shuffle partitions. (6) Use built-in functions over UDFs. (7) Cache reused DataFrames. (8) Compact small files. (9) Right-size executors. (10) Measure before/after. **Q711 — What is the difference between executor memory and driver memory?** **A711** — Driver: orchestrates the job, holds the DAG and results of collect(). Executor: runs tasks, holds cached data and shuffle. OOM on driver = collect too much. OOM on executor = partition too large, cache too much, or skew. **Q712 — What is memory overhead in Spark?** **A712** — Off-heap memory for JVM overhead, Python workers, and internal structures. `spark.executor.memoryOverhead` (default 10% of executor memory, min 384 MB). Too low causes container kills (exit code 137). **Q713 — What is GC tuning in Spark?** **A713** — Use G1GC (default in modern Spark), tune `spark.executor.extraJavaOptions`. Reduce object creation (use primitives, avoid unnecessary boxing). Monitor GC time in Spark UI. Excessive GC = memory pressure or too many small objects. **Q714 — What is the difference between `spark.sql.shuffle.partitions` and `spark.default.parallelism`?** **A714** — `spark.sql.shuffle.partitions` (default 200) controls partitions after a shuffle in SQL/DataFrame operations. `spark.default.parallelism` controls partitions for RDD operations. Set both based on data size and core count. AQE can auto-tune shuffle partitions. **Q715 — How do you size a Spark cluster?** **A715** — (1) Estimate data size per partition. (2) Partitions ≈ 2–4× total cores. (3) Executor memory = max partition size × safety factor + overhead. (4) Cores per executor = 4–5. (5) Number of executors = total cores / cores per executor. (6) Test and adjust. (7) Monitor spill and GC. **Q716 — How do you optimize a Redshift query?** **A716** — (1) Choose the right distribution key (join column). (2) Choose the right sort key (filter column). (3) VACUUM and ANALYZE. (4) Use COPY, not INSERT. (5) Compress columns. (6) Use WLM to manage concurrency. (7) Use materialized views for repeated queries. (8) Use Spectrum for cold data. (9) Check `SVL_QUERY_SUMMARY` and `SVL_QUERY_REPORT` for bottlenecks. **Q717 — How do you optimize Snowflake cost?** **A717** — (1) Right-size warehouses (start small, scale up). (2) Auto-suspend after short idle. (3) Use multi-cluster only when needed. (4) Cluster large tables on common filters. (5) Use Time Travel retention appropriately. (6) Use transient tables for staging. (7) Use resource monitors. (8) Avoid SELECT *. (9) Use `RESULT_SCAN` to reuse query results. **Q718 — How do you optimize BigQuery cost?** **A718** — (1) Partition tables on date. (2) Cluster on filter columns. (3) Select only needed columns. (4) Use `LIMIT` carefully (doesn't reduce scan). (5) Materialize repeated intermediate results. (6) Use `_TABLE_SUFFIX` to filter wildcard tables. (7) Use BI Engine for dashboards. (8) Consider flat-rate for predictable workloads. **Q719 — What is the difference between a full table scan and an index seek?** **A719** — Full scan reads every row/page of the table. Index seek navigates the B-tree to find matching rows directly. Seek is much faster for selective predicates. Scans are sometimes cheaper for large result sets. The optimizer chooses based on statistics. **Q720 — What is a covering index and when would you use one?** **A720** — An index that includes all columns needed by a query (key columns + INCLUDE columns), so the database doesn't need to touch the base table. Use when a query is read-heavy, the columns are stable, and the performance gain justifies the write cost and storage. --- ## SECTION 42 — FILE FORMATS & COMPRESSION **Q721 — What is the difference between Parquet and ORC?** **A721** — Both are columnar, compressed, and support predicate pushdown. Parquet is more common in the Spark/Databricks ecosystem; ORC is more common in Hive/Presto. Parquet has better nested-type support; ORC has better predicate pushdown for some engines. Choose based on ecosystem. **Q722 — What is the difference between Avro and Parquet?** **A722** — Avro: row-based, schema-embedded, good for write-heavy and streaming (Kafka). Parquet: columnar, good for read-heavy analytics. Avro is often the ingestion format; Parquet is the analytics format. **Q723 — What is the difference between Snappy, Gzip, and Zstd?** **A723** — Snappy: fast, low compression, good for intermediate data. Gzip: slower, high compression, good for archival. Zstd: fast and high compression, increasingly the default. Choose by write vs read vs storage trade-off. **Q724 — What is the difference between JSON, JSON Lines, and BSON?** **A724** — JSON: standard, nested, verbose. JSON Lines: one JSON object per line — streamable, splittable. BSON: binary JSON, used by MongoDB. JSON Lines is preferred for data pipelines (streamable, splittable, one bad line doesn't break the file). **Q725 — What is the difference between CSV and TSV?** **A725** — Delimiter: comma vs tab. TSV is safer when data contains commas. Both are row-based, weakly typed, and require escaping for embedded delimiters. Prefer Parquet for analytics. **Q726 — What is predicate pushdown?** **A726** — A query engine pushes filter conditions down to the storage layer, so only matching data is read. Parquet and ORC support this via row-group statistics (min/max). Delta and Iceberg extend it with table statistics. Huge performance win. **Q727 — What is column pruning?** **A727** — Only reading the columns needed by a query, possible with columnar formats. Reduces I/O and improves performance. Parquet and ORC support it natively. **Q728 — What is the difference between a row group and a page in Parquet?** **A728** — A row group is a horizontal chunk of rows (default 128 MB) with column chunks inside. A page is the smallest unit within a column chunk. Statistics exist at row group and page level. Smaller row groups = better pruning but more overhead. **Q729 — What is a small file problem?** **A729** — Many small files (KB instead of MB) cause: slow listing, high metadata overhead, poor parallelism, and inefficient scans. Solutions: compaction (OPTIMIZE, MERGE), controlling output partitions, and using table formats that handle small files. **Q730 — What is file compaction?** **A730** — Combining many small files into fewer large ones. In Delta: `OPTIMIZE`. In Iceberg: `rewriteDataFiles`. In Spark: `coalesce` or `repartition` before write. Improves read performance and reduces metadata overhead. --- ## SECTION 43 — PARTITIONING & CLUSTERING STRATEGIES **Q731 — What is the difference between partitioning and bucketing?** **A731** — Partitioning: divides data into directories by column values (e.g., date). Bucketing: hashes rows into a fixed number of files by a column (e.g., user_id). Partitioning is for pruning; bucketing is for joins and sampling. **Q732 — When would you use partitioning?** **A732** — When queries frequently filter on a low-cardinality column (date, region, country). Partitioning reduces scan. Avoid over-partitioning (millions of tiny partitions). Target 1 GB+ per partition. **Q733 — When would you use bucketing?** **A733** — When joining two large tables on the bucket key, or when sampling. Bucketing co-locates join keys, avoiding a shuffle. Requires the same bucket count and key on both sides. **Q734 — What is a partition key?** **A734** — The column(s) used to partition data. Choose based on filter patterns. Date is most common. Avoid high-cardinality columns (user_id) — use bucketing instead. **Q735 — What is a composite partition key?** **A735** — Partitioning by multiple columns, e.g., year/month/day or region/date. Enables finer pruning but can create many small partitions. Use when queries filter on multiple dimensions. **Q736 — What is dynamic partition overwrite?** **A736** — A Spark write mode that overwrites only the partitions present in the new data, leaving other partitions intact. `spark.sql.sources.partitionOverwriteMode = dynamic`. Enables idempotent partition-level reruns. **Q737 — What is partition pruning?** **A737** — A query engine skips partitions that don't match the filter. Requires the filter to be on the partition column and the catalog to know the partitions. Essential for performance on large tables. **Q738 — What is a hidden partition?** **A738** — A partition column that is not part of the table schema but is derived (e.g., date from a timestamp). Iceberg supports hidden partitioning, which simplifies queries and prevents mistakes. Users query the source column; the engine prunes. **Q739 — What is Z-ordering?** **A739** — A technique in Delta Lake that co-locates related data in the same files by multiple columns, improving data skipping. `OPTIMIZE table ZORDER BY (col1, col2)`. Use on columns frequently used in filters. Not the same as partitioning. **Q740 — What is liquid clustering?** **A740** — A Databricks feature that replaces static partitioning and Z-ordering with adaptive clustering keys that can change without rewriting data. Simplifies table design and adapts to query patterns. --- ## SECTION 44 — DATA SECURITY DEEP DIVE (SNOWFLAKE/CLOUD) **Q741 — What is dynamic data masking?** **A741** — Masking at query time based on the user's role. Snowflake: masking policies. BigQuery: policy tags. Redshift: dynamic data masking. The underlying data is unchanged; only the query result is masked. Different from static masking (permanent). **Q742 — What is tokenization?** **A742** — Replacing sensitive data with a non-sensitive token that maps back via a secure vault. The original is stored in the vault; the token is used in analytics. Preserves referential integrity (same input → same token) while removing PII from the data warehouse. **Q743 — What is format-preserving encryption (FPE)?** **A743** — Encryption that preserves the format (length, character set) of the original, so downstream systems don't break. Example: a 16-digit credit card number becomes another 16-digit number. Reversible with the key. Different from hashing (irreversible). **Q744 — What is a data access audit?** **A744** — Tracking who accessed what data and when. CloudTrail (AWS), Audit Logs (Azure), Access History (Snowflake), INFORMATION_SCHEMA.JOBS (BigQuery). Essential for compliance and incident investigation. **Q745 — What is the principle of least privilege in data?** **A745** — Grant only the access needed for a role to perform its function. Use role-based access control (RBAC), column-level security, row-level security, and just-in-time access. Review periodically. **Q746 — What is a data classification?** **A746** — Categorizing data by sensitivity: public, internal, confidential, restricted, PII, PHI, PCI. Drives access policies, retention, and masking. Often automated with scanners (Macie, Purview, Snowflake classification). **Q747 — What is a data residency requirement?** **A747** — Data must remain within a specific geographic region (e.g., EU data stays in EU). Implemented via region-restricted storage, processing, and replication. Conflicts with global analytics — resolve with regional warehouses and aggregated cross-region views. **Q748 — What is the GDPR right to erasure?** **A748** — Individuals can request deletion of their personal data. Requires pipelines to support deletion propagation (hard delete or crypto-shredding). Complicates immutable Bronze — use tombstone records or crypto-shredding (delete the key). **Q749 — What is crypto-shredding?** **A749** — Deleting the encryption key instead of the data, making the data unrecoverable. Satisfies erasure requirements without rewriting immutable storage. Requires per-subject keys. Trade-off: key management complexity. **Q750 — What is the difference between encryption at rest and in transit?** **A750** — At rest: data on disk is encrypted (KMS, TDE). In transit: data on the network is encrypted (TLS, VPN). Both are required for compliance. Encryption at rest protects against physical theft; in transit protects against interception. --- ## SECTION 45 — MLOps & ML PIPELINE TESTING (ON YOUR CV — SCIKIT-LEARN) **Q751 — What is MLOps?** **A751** — Machine Learning Operations — the practices and tools for deploying, monitoring, and maintaining ML models in production. Includes data versioning, feature stores, model registries, CI/CD for models, monitoring for drift, and retraining. **Q752 — How do you test an ML pipeline?** **A752** — (1) Data validation: schema, distribution, nulls, outliers. (2) Feature engineering tests: correctness, leakage. (3) Model training tests: reproducibility (same seed → same model), metric thresholds. (4) Model evaluation: holdout, cross-validation, bias. (5) Inference tests: latency, correctness, edge cases. (6) Monitoring: drift, performance decay. **Q753 — What is data leakage?** **A753** — Using information from the target or future data in training, leading to artificially high performance that doesn't generalize. Examples: normalizing before splitting, using future values in a time series, target encoding without cross-validation. Test: compare offline vs online performance. **Q754 — What is model drift?** **A754** — The model's performance degrades over time because the data distribution changes (data drift) or the relationship between features and target changes (concept drift). Monitor prediction distribution, feature distribution, and business metrics. Retrain when drift exceeds a threshold. **Q755 — What is a feature store?** **A755** — A centralized repository for features used in ML, with consistency between training and serving. Provides offline storage (for training) and online storage (for low-latency serving). Examples: Feast, Tecton, Databricks Feature Store. Prevents training-serving skew. **Q756 — What is training-serving skew?** **A756** — A mismatch between the data/features used in training and those used in serving, causing the model to perform worse in production. Prevented with a feature store and shared transformation code. **Q757 — What is A/B testing for models?** **A757** — Running two model versions simultaneously on different user segments and comparing business metrics. Requires randomization, sufficient sample size, and a clear success metric. More reliable than offline evaluation. **Q758 — What is a model registry?** **A758** — A catalog of model versions with metadata, metrics, and stage (staging/production/archived). Examples: MLflow Model Registry, SageMaker Model Registry. Enables reproducibility, rollback, and governance. **Q759 — How do you monitor an ML model in production?** **A759** — (1) Prediction distribution. (2) Feature distribution (drift). (3) Ground-truth performance (when labels arrive). (4) Business metrics. (5) Latency and error rate. (6) Bias/fairness metrics. Alert on drift and decay. **Q760 — What is explainability in ML?** **A760** — Understanding why a model made a prediction. Techniques: SHAP, LIME, feature importance, partial dependence. Required for regulated domains (credit, healthcare). Trade-off with model complexity. --- ## SECTION 46 — DATAOPS & ADVANCED OPERATIONS **Q761 — What is DataOps?** **A761** — Applying DevOps principles to data: version control, CI/CD, automated testing, monitoring, collaboration, and continuous improvement. Aims to reduce the time from data change to business value while maintaining quality. **Q762 — What is a data pipeline SLA?** **A762** — A commitment on pipeline performance: freshness (data available by X), completeness (all expected rows), accuracy (within tolerance), and availability (uptime). Measured and reported. Breaches trigger incidents. **Q763 — What is a data pipeline SLO?** **A763** — An internal target for the SLA, e.g., "99% of daily loads complete by 06:00." The SLA is what the business is promised; the SLO is the engineering target. SLOs are stricter than SLAs to provide a buffer. **Q764 — What is a data pipeline error budget?** **A764** — The allowable amount of failure or downtime before the SLA is breached. E.g., if the SLO is 99.9% freshness, the error budget is 0.1% of days. When the budget is exhausted, prioritize reliability over new features. **Q765 — What is a data pipeline runbook?** **A765** — A document describing how to detect, diagnose, mitigate, and escalate a specific incident. Includes symptoms, checks, commands, contacts, and rollback steps. Written before the incident, tested periodically. **Q766 — What is a data pipeline postmortem?** **A766** — A blameless review after an incident: timeline, impact, root cause, what went well, what went wrong, and tracked preventive actions with owners and dates. Focus on system conditions, not individuals. **Q767 — What is a data pipeline game day?** **A767** — A planned exercise to test incident response: inject a failure (kill a job, corrupt data, block a network) and observe how the team detects, responds, and recovers. Identifies gaps before real incidents. **Q768 — What is a data pipeline chaos experiment?** **A768** — A controlled injection of failure to test resilience: kill a pod, add latency, drop packets, corrupt data. Based on chaos engineering principles. Requires monitoring, rollback, and a hypothesis. **Q769 — What is a data pipeline canary?** **A769** — Releasing new pipeline logic to a small subset of data first, comparing outputs to the old logic, then rolling out. Reduces risk. Requires a comparison framework and rollback plan. **Q770 — What is a data pipeline blue-green deployment?** **A770** — Two identical environments: blue (live) and green (new). Deploy to green, validate, switch. Instant rollback. For pipelines, the "switch" is a pointer change or a table swap. **Q771 — What is a data pipeline feature flag?** **A771** — A configuration toggle that enables/disables new logic without redeploying. Useful for gradual rollout and quick disable. For pipelines, use flags to switch between old and new transformation logic. **Q772 — What is a data pipeline rollback?** **A772** — Reverting to a known-good state: previous code version, previous config, previous data. Code rollback alone is insufficient if bad data was written — you need data rollback or replay from a known-good source. --- ## SECTION 47 — COMPANY-SPECIFIC INTERVIEW PATTERNS **Q773 — How does XXX interview for data roles?** **A773** — Typically 2–3 rounds: technical (SQL, Python, ETL concepts, project deep-dive), managerial (behavioral, situation), HR. Focus on your project, SQL queries, and ETL fundamentals. May include a written test. Expect questions on your specific project and the technologies used. **Q774 — How does TCS interview for data roles?** **A774** — Technical round (SQL, ETL, project), managerial, HR. Often uses a standard question bank. Emphasizes communication and clarity. May ask about your current project in detail. **Q775 — How does Accenture interview for data roles?** **A775** — Similar to XXX/TCS, with more emphasis on consulting skills and client-facing communication. May include case-study-style questions. Technical depth varies by role level. **Q776 — How does Amazon interview for data roles?** **A776** — Bar-raiser process: 4–5 rounds including a bar-raiser (senior interviewer who can veto). Heavy on leadership principles (STAR format), SQL, and system design. Expect questions on scale, ownership, and customer obsession. Writing code on a whiteboard. **Q777 — How does Microsoft interview for data roles?** **A777** — 3–5 rounds: technical (SQL, coding, design), behavioral, hiring manager. Focus on problem-solving, collaboration, and growth mindset. Azure-specific questions if the role is Azure-focused. **Q778 — How do product companies (Google, Meta) interview for data roles?** **A778** — 4–5 rounds: coding (Python/SQL), data modeling, system design, behavioral. Heavy on scale, trade-offs, and first-principles thinking. Whiteboard coding. SQL questions are complex (window functions, optimization). **Q779 — What should you do in the first 5 minutes of a technical interview?** **A779** — (1) Greet, confirm the format. (2) Listen carefully to the question. (3) Ask clarifying questions. (4) State your approach before coding. (5) Think aloud. (6) Test with an example. (7) Discuss trade-offs. (8) Ask for feedback. **Q780 — What should you do if you don't know the answer?** **A780** — (1) Be honest: "I haven't used that specific tool, but here is how I would approach it." (2) Relate to something you do know. (3) Show your reasoning. (4) Ask for a hint if appropriate. (5) Never bluff — interviewers see through it. **Q781 — What are the most common mistakes in a technical interview?** **A781** — (1) Bluffing. (2) Not asking clarifying questions. (3) Jumping to code without a plan. (4) Ignoring edge cases. (5) Not testing. (6) Poor communication. (7) Arguing with the interviewer. (8) Not admitting when you're stuck. (9) Not preparing project deep-dives. (10) Not asking questions at the end. **Q782 — How do you handle a question about a tool you don't know?** **A782** — Use the gap template: "I haven't used [tool] in a project. The validation approach is the same: counts, aggregates, row-level diff, business rules. In [tool] I would use [equivalent feature]. I have done the same in [tool you know], and I ramp up quickly." **Q783 — How do you handle a question about a project you didn't work on?** **A783** — Be honest: "I didn't work on that project, but based on my experience with [similar project], I would approach it by [method]." Never claim credit you don't deserve. **Q784 — What is the best way to prepare for a project deep-dive?** **A784** — (1) Know your CV line by line. (2) For each project, prepare: sources, formats, volumes, layers, transformations, DQ controls, failures, fixes. (3) Prepare 3 STAR stories. (4) Anticipate 5 "why" questions for each decision. (5) Be ready to write code for any tool on your CV. **Q785 — What should you do after the interview?** **A785** — (1) Send a thank-you note within 24 hours. (2) Reflect on what went well and what didn't. (3) Note questions you struggled with and prepare better answers. (4) Follow up if no response within the stated timeline. --- ## SECTION 48 — ADVANCED ARCHITECTURE PATTERNS **Q786 — What is the Lambda architecture?** **A786** — A data architecture with three layers: batch (accurate, slow), speed (real-time, approximate), and serving (merges both). Provides both accuracy and low latency but requires maintaining two codebases. Largely superseded by the lakehouse and streaming-first designs. **Q787 — What is the Kappa architecture?** **A787** — A streaming-first architecture where all data is processed as a stream. Batch is a special case of streaming (replay the stream). Simpler than Lambda (one codebase) but requires reliable, replayable streaming. **Q788 — What is the medallion architecture?** **A788** — Bronze (raw), Silver (cleansed), Gold (curated) layers. Databricks terminology. Each layer adds quality and structure. Enables replay from Bronze, isolation of concerns, and clear contracts. **Q789 — What is a data vault?** **A789** — A modeling approach with Hubs (business keys), Links (relationships), and Satellites (descriptive attributes and history). Highly auditable and scalable, good for integrating many sources. More complex than star schema — used in large enterprises. **Q790 — What is a data mesh?** **A790** — A decentralized approach where domain teams own their data products, publish them with contracts, and a central platform provides self-service infrastructure and governance. Contrasts with a central data team owning everything. **Q791 — What is a data fabric?** **A791** — A metadata-driven approach that connects disparate data sources with automated integration, governance, and access. Emphasizes virtualization and active metadata over physical consolidation. **Q792 — What is a data product?** **A792** — A curated, governed dataset with a defined owner, SLA, schema, quality contract, and consumers. Treated as a product, not a byproduct of a pipeline. Gold tables in a well-run platform are data products. **Q793 — What is a data contract?** **A793** — A versioned agreement between producer and consumer: schema, semantics, keys, nullability, freshness, compatibility rules, ownership, and SLA. Enforced in CI/CD (schema gate) and monitored in production (freshness, schema drift). **Q794 — What is a data SLA vs SLO vs SLI?** **A794** — SLI (Service Level Indicator): a measured metric (freshness, completeness, latency). SLO (Objective): the target for the SLI. SLA (Agreement): a commitment with consequences if the SLO is missed. **Q795 — What is a data quality scorecard?** **A795** — A dashboard showing DQ metrics per dataset: completeness %, validity %, uniqueness %, freshness, reconciliation pass rate, defect count. Used to track trends and prioritize remediation. **Q796 — What is a data observability platform?** **A796** — A tool that monitors data health (freshness, volume, schema, distribution, lineage) and alerts on anomalies. Examples: Monte Carlo, Bigeye, Soda, Great Expectations. Complements operational monitoring. **Q797 — What is a data reliability engineer?** **A797** — A role focused on the reliability of data pipelines and data quality: SLOs, monitoring, incident response, root cause analysis, and prevention. Combines data engineering with SRE principles. **Q798 — What is the difference between data governance and data management?** **A798** — Governance: the policies, standards, and roles that ensure data is accurate, available, secure, and compliant. Management: the execution — catalog, lineage, quality, access, retention, and stewardship. Governance sets the rules; management follows them. **Q799 — What is a data catalog?** **A799** — A metadata repository describing datasets, schemas, locations, owners, lineage, and quality. Enables discovery, governance, and self-service. Examples: Glue Data Catalog, Unity Catalog, Purview, Datahub, Alation. **Q800 — What is the future of data engineering?** **A800** — Trends: (1) Lakehouse consolidation (Delta, Iceberg). (2) Streaming-first (Kafka, Flink). (3) AI/ML integration (feature stores, MLOps). (4) Data mesh and decentralization. (5) Active metadata and observability. (6) Cost optimization (FinOps). (7) Governance and privacy by design. (8) Serverless and managed services. --- ## FINAL COUNT | Section | Topic | Q&As | |---|---|---| | 1–21 | Original bank | 445 | | 22–35 | First gap additions | 185 | | 36–48 | Second gap additions | 170 | | **Total** | | **800** | --- ## WHAT WAS STILL MISSING — SUMMARY | Gap Area | Why It Matters | New Section | |---|---|---| | Snowflake deep dive | JD mentions; CV thin | 36 | | BigQuery deep dive | JD mentions; CV thin | 37 | | Redshift deep dive | JD mentions; CV says exposure | 38 | | dbt | Common in modern stacks | 39 | | Distributed systems concepts | Senior-level questions | 40 | | Performance tuning deep dive | Common senior questions | 41 | | File formats & compression | Common data engineering questions | 42 | | Partitioning & clustering strategies | Common design questions | 43 | | Data security deep dive (Snowflake/cloud) | Governance and compliance | 44 | | MLOps & ML pipeline testing | CV mentions Scikit-learn, MLflow | 45 | | DataOps & advanced operations | Senior-level expectations | 46 | | Company-specific interview patterns | Tailor preparation | 47 | | Advanced architecture patterns | Senior-level design questions | 48 | --- **Total Q&A bank now: 800 questions.** This covers every line on your CV, every domain in both architecture diagrams, every failure area, every common interview pattern, and the specific tools and concepts that come up at XXX, TCS, Accenture, Amazon, Microsoft, and product firms. Use it as a complete reference; prioritize by the job description and your own hands-on experience. --- Source file: Pasted text (4).txt # Still Missing — Additional Q&A Bank (Q801–Q950) After reviewing the 800-question bank against the combined uploaded files, your CV, and real interview patterns at XXX/TCS/Accenture/Amazon/Microsoft, the following **150 questions are still missing**. These cover lab tools mentioned in your CV (Hadoop, Hive, Iceberg, Flink, Trino, Prometheus/Grafana), advanced Python, practical data quality frameworks, anti-patterns, system design frameworks, and career/communication skills. --- ## SECTION 49 — HADOOP ECOSYSTEM (ON YOUR CV — LAB) **Q801 — What is HDFS?** **A801** — Hadoop Distributed File System: stores data in blocks (default 128 MB) across DataNodes, with metadata on the NameNode. Replication (default 3) provides fault tolerance. Write-once, read-many. Not suitable for small files or random updates. **Q802 — What is YARN?** **A802** — Yet Another Resource Negotiator: the resource manager for Hadoop. Allocates CPU and memory to applications. Components: ResourceManager (global), NodeManager (per node), ApplicationMaster (per app). Enables multiple processing engines (MapReduce, Spark, Tez) on the same cluster. **Q803 — What is the NameNode vs DataNode?** **A803** — NameNode: stores metadata (file names, permissions, block locations). Single point of failure unless HA is configured. DataNode: stores actual data blocks, sends heartbeats. Secondary NameNode is not a backup — it checkpoints the edit log. **Q804 — What is a block in HDFS?** **A804** — The smallest unit of storage (default 128 MB in Hadoop 2+). Files are split into blocks, replicated across nodes. Small files waste NameNode memory (one metadata entry per file, regardless of size). **Q805 — What is the small file problem in HDFS?** **A805** — Many small files overload the NameNode (memory) and slow MapReduce/Spark (one task per file). Solutions: HAR (Hadoop Archive), sequence files, CombineFileInputFormat, or compact files before ingestion. **Q806 — What is MapReduce?** **A806** — A programming model with two phases: Map (process key-value pairs in parallel) and Reduce (aggregate by key). Between them, a shuffle sorts and partitions. Writes to disk between stages — slower than Spark but more stable for very large jobs. **Q807 — What is a combiner in MapReduce?** **A807** — A mini-reducer that runs on the map output before the shuffle. Reduces network traffic by pre-aggregating. Must be commutative and associative (e.g., sum, count). Not always applicable. **Q808 — What is Hive?** **A808** — A SQL-like interface on Hadoop. HiveQL translates to MapReduce, Tez, or Spark jobs. Stores metadata in a metastore (usually MySQL/Postgres). Tables are directories in HDFS; partitions are subdirectories. Not for OLTP — for batch analytics. **Q809 — What is the Hive Metastore?** **A809** — A database (MySQL/Postgres) storing table definitions, schemas, partitions, and locations. Shared by Hive, Spark, Presto/Trino, and other engines. The Glue Data Catalog is AWS's managed metastore. **Q810 — What is the difference between internal and external tables in Hive?** **A810** — Internal (managed): Hive owns the data; dropping the table deletes the data. External: Hive owns only metadata; dropping the table leaves the data. Use external for data shared with other tools or for raw data you don't want to lose. **Q811 — What is partitioning in Hive?** **A811** — Dividing a table into subdirectories by column values (e.g., date=2026-10-01). Queries with a partition filter scan only relevant directories. Over-partitioning creates small files. Choose low-cardinality columns. **Q812 — What is bucketing in Hive?** **A812** — Hashing rows into a fixed number of files by a column. Enables efficient joins (bucket-to-bucket) and sampling. Requires CLUSTERED BY and the same bucket count on both sides. Different from partitioning. **Q813 — What is the difference between Hive and Spark SQL?** **A813** — Hive: SQL on Hadoop, originally MapReduce, later Tez. Spark SQL: SQL on the Spark engine (in-memory, DAG). Spark is faster for iterative work; Hive is more mature for batch. Both share the Hive Metastore. **Q814 — What is Tez?** **A814** — An execution engine for Hive and Pig that builds a DAG of tasks instead of sequential MapReduce jobs. Faster than MapReduce because it avoids intermediate disk writes. Superseded by Spark in many cases. **Q815 — What is HBase?** **A815** — A column-family NoSQL database on HDFS. Supports random read/write at scale (unlike HDFS). Used for real-time lookups on big data. Row key design is critical for performance. Not a replacement for a relational database. --- ## SECTION 50 — APACHE ICEBERG (ON YOUR CV — LAB) **Q816 — What is Apache Iceberg?** **A816** — An open table format for huge analytic datasets. Adds ACID transactions, schema evolution, partition evolution, and time travel to files on object storage (S3, ADLS, GCS). Engine-neutral (Spark, Trino, Flink, Hive, Presto). **Q817 — How does Iceberg differ from Hive tables?** **A817** — Iceberg tracks files in a metadata tree (manifest files, manifest lists, metadata.json), not directories. This enables atomic commits, hidden partitioning, schema evolution, and time travel. Hive relies on directory listing — slow and non-atomic. **Q818 — What is a snapshot in Iceberg?** **A818** — A point-in-time view of the table's data files. Every write creates a new snapshot. Time travel queries a specific snapshot by ID or timestamp. Old snapshots can be expired to reclaim storage. **Q819 — What is hidden partitioning in Iceberg?** **A819** — Partitioning is defined by a transform on a column (e.g., `days(ts)`), not by the column value itself. Users query the source column (`ts`); Iceberg prunes partitions automatically. Avoids the "partition column must be in the query" problem. **Q820 — What is partition evolution in Iceberg?** **A820** — Changing the partition spec without rewriting existing data. Old data uses the old spec; new data uses the new spec. Queries work across both. Not possible with Hive tables. **Q821 — What is schema evolution in Iceberg?** **A821** — Adding, dropping, renaming, or reordering columns safely. Column IDs (not names) track fields, so renames don't break data. Type promotion (int → long) is supported. Dropping a column keeps the data for time travel. **Q822 — What is a manifest file in Iceberg?** **A822** — A file listing data files with their partition values, column stats (min/max, null count), and file path. Iceberg uses manifests for pruning — the engine reads metadata, not data, to decide which files to scan. **Q823 — What is a manifest list?** **A823** — A file listing manifest files for a snapshot. The metadata.json points to the current manifest list. This tree structure enables fast planning without listing the data directory. **Q824 — What is the difference between Iceberg and Delta Lake?** **A824** — Both provide ACID, schema evolution, and time travel on object storage. Delta is Spark/Databricks-centric; Iceberg is engine-neutral (Spark, Trino, Flink, Hive). Delta has a transaction log (JSON + Parquet); Iceberg has a metadata tree. Choose based on ecosystem. **Q825 — What is the difference between copy-on-write and merge-on-read in Iceberg?** **A825** — Copy-on-write: updates rewrite affected data files (slower write, faster read). Merge-on-read: updates write delete files and new data files; readers merge them (faster write, slower read). Choose based on read/write ratio. **Q826 — How do you validate an Iceberg table?** **A826** — (1) Query `table.snapshots` metadata for history. (2) Compare current snapshot vs prior with time travel. (3) Check `table.files` for file-level stats. (4) Validate partition spec and schema. (5) Reconcile counts and sums. (6) Expire old snapshots to reclaim storage. **Q827 — What is the Iceberg catalog?** **A827** — The component that tracks the current metadata pointer for each table. Options: Hive Metastore, AWS Glue, Nessie, JDBC, REST. The catalog is the source of truth for "which metadata.json is current." --- ## SECTION 51 — APACHE FLINK (ON YOUR CV — LAB) **Q828 — What is Apache Flink?** **A828** — A distributed stream processing engine with true event-time processing, exactly-once state consistency, and low latency. Handles both streaming and batch (batch is a bounded stream). Competes with Spark Structured Streaming. **Q829 — What is the difference between Flink and Spark Structured Streaming?** **A829** — Flink is streaming-first (batch is a special case); Spark is batch-first (streaming is micro-batch). Flink has lower latency (milliseconds), true event-time processing, and more sophisticated state management. Spark is easier to adopt if you already use Spark. **Q830 — What is event time vs processing time in Flink?** **A830** — Event time: when the event occurred (from the event's timestamp). Processing time: when the engine processes it. Flink supports both, plus ingestion time. Event time enables correct results despite out-of-order events. **Q831 — What is a watermark in Flink?** **A831** — A signal that no more events with timestamp ≤ T will arrive. Enables event-time windows to close. Watermarks are generated by the source or a watermark strategy. Late events after the watermark are handled by allowed lateness or side outputs. **Q832 — What is a window in Flink?** **A832** — A grouping of events over time or count. Types: tumbling (fixed, non-overlapping), sliding (fixed, overlapping), session (gap-based), global (all events until a trigger). Applied to keyed streams. **Q833 — What is Flink state?** **A833** — Data retained across events for a key (e.g., running count, last value). Types: keyed state (per key), operator state (per operator). Backends: HashMapStateBackend (memory), EmbeddedRocksDBStateBackend (disk). Checkpointed for fault tolerance. **Q834 — What is a checkpoint in Flink?** **A834** — A consistent snapshot of all operator state, written to durable storage (S3, HDFS). On failure, Flink restores from the last checkpoint. Barriers flow through the DAG to align checkpoints. Exactly-once is achieved with checkpointing + transactional sinks. **Q835 — What is a savepoint in Flink?** **A835** — A manual checkpoint used for upgrades, scaling, or migration. Unlike checkpoints (automatic, for recovery), savepoints are triggered by the user and preserved. Essential for zero-downtime deployments. **Q836 — What is the difference between exactly-once and at-least-once in Flink?** **A836** — At-least-once: events may be reprocessed on failure. Exactly-once: state is consistent, and sinks commit transactionally (two-phase commit). Exactly-once requires a transactional sink (Kafka, file system with atomic rename). **Q837 — What is backpressure in Flink?** **A837** — When a downstream operator is slower than upstream, backpressure propagates. Flink handles it via credit-based flow control. Persistent backpressure indicates a bottleneck — check the Flink UI for the busy operator. **Q838 — What is a keyed stream in Flink?** **A838** — A stream partitioned by a key, enabling keyed state and windowing. Operations like `keyBy`, `reduce`, `aggregate`, and `window` require a keyed stream. Key choice determines parallelism and state distribution. --- ## SECTION 52 — TRINO / PRESTO (ON YOUR CV — LAB) **Q839 — What is Trino (formerly Presto)?** **A839** — A distributed SQL query engine for federated queries across heterogeneous sources (Hive, Iceberg, Kafka, RDBMS, S3). MPP architecture with a coordinator and workers. No storage of its own — queries data where it lives. **Q840 — What is the difference between Trino and Hive?** **A840** — Trino is a query engine (no storage); Hive is a SQL layer on Hadoop with its own metastore. Trino can query Hive tables via the Hive connector. Trino is faster for interactive queries; Hive is for batch ETL. **Q841 — What is the Trino coordinator vs worker?** **A841** — Coordinator: parses SQL, plans the query, schedules tasks, manages workers. Worker: executes tasks, reads data, shuffles. Clients connect to the coordinator only. **Q842 — What is a connector in Trino?** **A842** — A plugin that lets Trino read/write a specific data source. Examples: Hive, Iceberg, Delta, Kafka, MySQL, PostgreSQL, MongoDB. Trino federates across connectors in a single query. **Q843 — What is the difference between Trino and Spark SQL?** **A843** — Trino is optimized for interactive, low-latency SQL on data where it lives. Spark SQL is optimized for large-scale ETL with its own compute and storage. Trino doesn't write to its own storage; Spark can. Use Trino for ad-hoc queries, Spark for pipelines. **Q844 — What is a catalog in Trino?** **A844** — A named configuration of a connector + metastore. Example: `hive` catalog pointing to the Hive Metastore, `iceberg` catalog pointing to Iceberg tables. Schemas and tables live inside catalogs. **Q845 — How do you validate data through Trino?** **A845** — Same approach as any SQL engine: counts, aggregates, anti-joins, hash comparison. Trino's strength is querying across sources in one query — useful for cross-platform reconciliation. --- ## SECTION 53 — PROMETHEUS & GRAFANA (ON YOUR CV — LAB) **Q846 — What is Prometheus?** **A846** — An open-source monitoring system that scrapes metrics from targets (pull model), stores them in a time-series database, and supports alerting via Alertmanager. Uses PromQL for queries. **Q847 — What is the pull vs push model?** **A847** — Pull: Prometheus scrapes targets at intervals (default 15s). Push: targets send metrics to a gateway (Pushgateway for short-lived jobs). Pull is preferred for long-running services; push for batch jobs. **Q848 — What is a metric in Prometheus?** **A848** — A named time series with labels. Types: Counter (monotonic), Gauge (up/down), Histogram (buckets), Summary (quantiles). Example: `http_requests_total{method="GET", status="200"}`. **Q849 — What is PromQL?** **A849** — Prometheus Query Language. Examples: `rate(http_requests_total[5m])` (per-second rate), `sum by (status) (rate(...))` (aggregate), `histogram_quantile(0.95, ...)` (p95). Used in dashboards and alerts. **Q850 — What is Alertmanager?** **A850** — Routes alerts from Prometheus to receivers (email, Slack, PagerDuty). Handles grouping, inhibition, and silencing. Configurable via YAML. **Q851 — What is Grafana?** **A851** — A visualization and dashboarding tool. Connects to Prometheus, Loki, Elasticsearch, SQL databases, and many other sources. Builds time-series panels, tables, heatmaps, and alerts. **Q852 — What is the difference between Prometheus and CloudWatch?** **A852** — Prometheus is open-source, self-hosted, pull-based, and multi-cloud. CloudWatch is AWS-managed, push-based, and AWS-native. Many teams use both: CloudWatch for AWS services, Prometheus for application metrics. **Q853 — What is a Grafana dashboard for a data pipeline?** **A853** — Panels for: job success/failure, duration, freshness, row counts, rejects, retries, consumer lag, DLQ depth, cost. One dashboard per pipeline or one overview with drill-downs. Alerts on thresholds. **Q854 — What is the difference between logging and metrics?** **A854** — Logs: detailed event records (text/JSON), high cardinality, expensive to store/query. Metrics: numeric time series, low cardinality, cheap to store/query, good for alerting. Use both: logs for investigation, metrics for monitoring. **Q855 — What is distributed tracing?** **A855** — Tracking a request across services with a trace ID. Tools: Jaeger, Zipkin, AWS X-Ray. For data pipelines, tracing shows how a record flows through ingestion, transformation, and serving — useful for root cause. --- ## SECTION 54 — ADVANCED PYTHON (SENIOR INTERVIEWS) **Q856 — What is a Python decorator with arguments?** **A856** — A decorator factory: a function that returns a decorator. Example: ```python def repeat(n): def deco(fn): def wrapper(*args, **kwargs): for _ in range(n): result = fn(*args, **kwargs) return result return wrapper return deco @repeat(3) def hello(): print("hi") ``` **Q857 — What is a generator expression vs list comprehension?** **A857** — List comprehension `[x*2 for x in range(10)]` builds the full list in memory. Generator expression `(x*2 for x in range(10))` yields lazily. Use generators for large data or when you only iterate once. **Q858 — What is `yield from`?** **A858** — Delegates to a sub-generator, yielding its values. Simplifies nested generators. Example: `yield from range(10)` instead of `for i in range(10): yield i`. **Q859 — What is a context manager and how do you write one?** **A859** — An object with `__enter__` and `__exit__` methods, used with `with`. Or use `@contextmanager` from `contextlib`: ```python from contextlib import contextmanager @contextmanager def timer(label): start = time.time() yield print(f"{label}: {time.time() - start:.2f}s") ``` **Q860 — What is the difference between `__init__` and `__new__`?** **A860** — `__new__` creates the instance (returns it); `__init__` initializes it. Rarely overridden. Used for immutable types, singletons, or metaclasses. **Q861 — What is a metaclass?** **A861** — A class of a class. Controls how classes are created. Rarely needed. Used in frameworks (Django ORM, SQLAlchemy) to add behavior to classes at definition time. **Q862 — What is the difference between `@staticmethod`, `@classmethod`, and instance methods?** **A862** — Instance method: takes `self`, operates on the instance. Class method: takes `cls`, operates on the class (alternative constructors). Static method: takes neither, just a function namespaced in the class. **Q863 — What is `functools.lru_cache`?** **A862** — A decorator that caches function results by arguments. Speeds up expensive pure functions. `@lru_cache(maxsize=128)`. Use `cache_info()` to monitor hits/misses. **Q864 — What is the difference between `deepcopy` and `copy`?** **A864** — `copy.copy` creates a shallow copy (nested objects are shared). `copy.deepcopy` recursively copies all nested objects. Use deepcopy when you need full independence. **Q865 — What is a dataclass?** **A865** — A decorator that auto-generates `__init__`, `__repr__`, `__eq__` from class attributes. Reduces boilerplate. Example: ```python from dataclasses import dataclass @dataclass class Trade: trade_id: str amount: float ccy: str = "USD" ``` **Q866 — What is type hinting?** **A866** — Annotating types for readability and static analysis. `def f(x: int) -> str:`. Not enforced at runtime (unless using pydantic or similar). Tools: mypy, pyright. **Q867 — What is `asyncio`?** **A867** — Python's async I/O framework. `async def` defines a coroutine; `await` yields control. Use for I/O-bound concurrency (HTTP calls, DB queries). Not for CPU-bound work (use multiprocessing). **Q868 — What is the GIL?** **A868** — Global Interpreter Lock: only one thread executes Python bytecode at a time. Limits CPU-bound multithreading. Workarounds: multiprocessing, C extensions (NumPy releases GIL), or async I/O for I/O-bound work. **Q869 — What is `__slots__`?** **A869** — A class attribute that restricts instance attributes, saving memory. `__slots__ = ('a', 'b')`. Useful for millions of small objects. Trade-off: no `__dict__`, no dynamic attributes. **Q870 — What is the difference between `is` and `==`?** **A870** — `is` checks identity (same object in memory); `==` checks equality (same value). Use `is` for `None`, `True`, `False`. Use `==` for value comparison. --- ## SECTION 55 — ADVANCED PYTHON TESTING (pytest) **Q871 — What is a pytest fixture?** **A871** — A reusable setup/teardown function. `@pytest.fixture` decorates it; tests request it by name. Scopes: function, class, module, session. `yield` for teardown. **Q872 — What is fixture scope?** **A872** — Function (default): new per test. Class: per class. Module: per module. Session: once per test run. Use session for expensive setup (DB connection). Use function for isolation. **Q873 — What is `pytest.mark.parametrize`?** **A873** — Runs a test multiple times with different inputs: ```python @pytest.mark.parametrize("x,expected", [(1, 2), (2, 4), (3, 6)]) def test_double(x, expected): assert x * 2 == expected ``` **Q874 — What is `monkeypatch`?** **A874** — A pytest fixture for temporarily modifying attributes, dicts, env vars, or sys.path. Useful for mocking. Example: `monkeypatch.setenv("DB_URL", "test")`. **Q875 — What is `mock.patch`?** **A875** — Replaces an object with a MagicMock for the duration of a test. `@patch('module.function')`. Use for external dependencies (APIs, DBs). Assert calls with `mock.assert_called_once_with(...)`. **Q876 — What is test coverage?** **A876** — The percentage of code lines executed by tests. Tools: `coverage.py`, `pytest-cov`. Target: high coverage on critical paths, not 100% everywhere. Coverage alone doesn't prove correctness. **Q877 — What is a flaky test?** **A877** — A test that passes or fails non-deterministically. Causes: timing, ordering, external dependencies, shared state. Fix: isolate state, mock external calls, use deterministic data. Plugin: `pytest-rerunfailures`. **Q878 — What is the difference between `assert` and `pytest.raises`?** **A878** — `assert` checks a condition. `pytest.raises(ValueError)` asserts that a block raises a specific exception: ```python with pytest.raises(ValueError, match="invalid"): parse("bad") ``` **Q879 — What is a test double?** **A879** — A generic term for a fake object used in testing: stub (returns canned data), mock (records calls), spy (wraps real object), fake (simplified implementation). Python's `unittest.mock` provides MagicMock. **Q880 — What is property-based testing?** **A880** — Generating random inputs to find edge cases. Library: `hypothesis`. Example: test that sorting is idempotent for any list. Complements example-based tests. --- ## SECTION 56 — DATA QUALITY FRAMEWORKS COMPARISON **Q881 — Compare Great Expectations, Soda, and Deequ.** **A881** — Great Expectations: Python-based, declarative expectations, data docs, pipeline gate. Soda: SQL-based checks (SodaCL), runs in the warehouse, good for non-Python teams. Deequ: Scala/Spark library (Amazon), scales with Spark, unit-test style. Choose by language, scale, and ecosystem. **Q882 — What is the difference between rule-based and ML-based DQ?** **A882** — Rule-based: explicit thresholds (not null, in range). Deterministic, explainable, but requires maintenance. ML-based: learns patterns and flags anomalies (volume, distribution). Adapts to changing data but can miss known rules and is harder to explain. Best: combine both. **Q883 — What is a DQ dimension vs a DQ rule vs a DQ check?** **A883** — Dimension: category (completeness, accuracy). Rule: a specific testable condition (trade_id not null). Check: the execution of a rule on a dataset at a point in time, producing a pass/fail. **Q884 — What is a DQ threshold vs a DQ tolerance?** **A884** — Threshold: the acceptable limit (e.g., null % < 0.1%). Tolerance: the allowed deviation from expected (e.g., sum within 0.01). Both are configured per rule and per dataset. **Q885 — How do you prioritize DQ rules?** **A885** — By business impact: financial correctness > operational metrics > cosmetic. By data criticality: Gold > Silver > Bronze. By change frequency: new pipelines, recent changes. By regulatory requirement: KYC/AML rules are non-negotiable. **Q886 — What is a DQ score?** **A886** — A composite metric for a dataset: weighted average of pass rates across rules. Used for dashboards and prioritization. Not a substitute for per-rule detail. **Q887 — What is DQ drift?** **A887** — A change in data distribution over time (e.g., null rate creeping up, a new category appearing). Detected by comparing profiles across days. Early warning of upstream issues. **Q888 — What is a DQ incident vs a DQ defect?** **A888** — Incident: a DQ failure detected in production (data is bad now). Defect: a flaw in the pipeline or rule that causes incidents. Incidents are symptoms; defects are causes. Track both. **Q889 — How do you measure DQ ROI?** **A889** — Cost of bad data (wrong decisions, rework, compliance fines) vs cost of DQ program (tools, people, compute). Hard to quantify exactly, but incidents and escaped defects are leading indicators. **Q890 — What is a DQ contract?** **A890** — A written agreement between producer and consumer: which DQ rules apply, thresholds, severity, and consequences (block publication, alert, or warn). Enforced in the pipeline and monitored. --- ## SECTION 57 — DATA ENGINEERING ANTI-PATTERNS **Q891 — What is the SELECT * anti-pattern?** **A891** — Selecting all columns when only a few are needed. Wastes I/O (columnar formats), breaks when schema changes, and hides intent. Always list columns explicitly. **Q892 — What is the NOT IN with NULL anti-pattern?** **A892** — `WHERE id NOT IN (SELECT id FROM t)` returns zero rows if the subquery has any NULL. Use `NOT EXISTS` or `LEFT ANTI JOIN` instead. **Q893 — What is the implicit cast anti-pattern?** **A893** — Comparing columns of different types (varchar vs nvarchar, int vs varchar) forces a cast, prevents index use, and can silently produce wrong results. Align types explicitly. **Q894 — What is the non-sargable predicate anti-pattern?** **A894** — Applying a function to an indexed column in a WHERE clause (`YEAR(date) = 2024`, `UPPER(name) = 'ABC'`). Prevents index use. Rewrite as a range or use a function-based index. **Q895 — What is the small files anti-pattern?** **A895** — Writing thousands of tiny files to a data lake. Slows listing, planning, and querying. Causes: over-partitioning, too many shuffle partitions, streaming micro-batches. Fix: compaction, coalesce, or tune partitions. **Q896 — What is the N+1 query anti-pattern?** **A896** — Executing one query per row in a loop, causing N+1 database round trips. Fix: batch the query, use a join, or use a set-based operation. **Q897 — What is the hardcoded credentials anti-pattern?** **A897** — Storing passwords, API keys, or connection strings in code, notebooks, or config files. Fix: use a secret manager, environment variables, or managed identity. Rotate regularly. **Q898 — What is the no-idempotency anti-pattern?** **A898** — A pipeline that duplicates or corrupts data when rerun. Causes: blind INSERT, no dedup, no partition scope. Fix: MERGE, partition overwrite, or delete+insert with deterministic keys. **Q899 — What is the no-DQ-gate anti-pattern?** **A899** — Publishing data to Gold without validation. Consumers see bad data. Fix: DQ checks at each layer, critical rules block publication, quarantine bad records. **Q900 — What is the no-observability anti-pattern?** **A900** — A pipeline that fails silently or succeeds with wrong data, with no alerts. Fix: metrics, logs, freshness checks, alerting, and runbooks. **Q901 — What is the monolithic pipeline anti-pattern?** **A901** — One giant job that does everything: ingest, transform, aggregate, serve. Hard to test, debug, and rerun. Fix: decompose into layers and small jobs with clear contracts. **Q902 — What is the manual-step anti-pattern?** **A902** — A pipeline that requires a human to run a script or approve a step. Doesn't scale, introduces errors. Fix: automate everything except genuine business approvals. **Q903 — What is the schema-on-read-everything anti-pattern?** **A903** — Treating all data as schemaless and parsing at query time. Slow, error-prone, no governance. Fix: schema-on-write for curated layers, schema-on-read for exploration. **Q904 — What is the copy-paste pipeline anti-pattern?** **A904** — Duplicating pipeline code for each dataset instead of parameterizing. Multiplies bugs. Fix: config-driven frameworks and reusable components. **Q905 — What is the no-backfill anti-pattern?** **A905** — A pipeline that only processes "today" and cannot reprocess history. Fix: parameterize by date range, support replay from immutable Bronze. --- ## SECTION 58 — SYSTEM DESIGN FRAMEWORK FOR DATA ENGINEERING **Q906 — How do you approach a data system design question?** **A906** — (1) Clarify requirements: sources, volume, latency, SLA, consumers, compliance. (2) Sketch the architecture: ingestion, storage, processing, serving. (3) Choose technologies based on requirements. (4) Define the data model and contracts. (5) Plan for reliability: idempotency, retries, DQ gates. (6) Plan for scale: partitioning, parallelism, cost. (7) Plan for operations: monitoring, alerting, runbooks. (8) Discuss trade-offs. **Q907 — Design a real-time fraud detection pipeline.** **A907** — Sources: transactions (Kafka), customer reference (CDC). Ingest to MSK with schema registry. Flink/Spark Streaming for enrichment (customer profile, velocity checks) and scoring (model). Feature store for online features. Sink: alerts (SNS) and case management (Aurora). Bronze/Silver/Gold in S3 for audit and model retraining. Latency target: < 1 second end-to-end. Idempotency by transaction ID. DQ rules: valid account, amount range, no duplicate. Compliance: PII masked, audit trail. **Q908 — Design a daily reporting platform for 1 TB/day.** **A908** — Sources: 50 source systems (RDBMS, files, APIs). Ingest via ADF/Glue into S3 Bronze (partitioned by date/source). Orchestrate with Step Functions/Airflow. Transform with Databricks/Spark into Silver (cleansed, conformed) and Gold (business aggregates). DQ gates with Great Expectations. Serve via Athena/Redshift/QuickSight. SLA: 06:00 IST freshness. Idempotent daily partitions. Backfill support. Cost: auto-scaling clusters, S3 lifecycle to Glacier after 90 days. **Q909 — Design a data lake for a bank with PII and regulatory requirements.** **A909** — Multi-account AWS: landing, raw, curated, governance. S3 with KMS encryption, bucket policies, VPC endpoints. Lake Formation for table/column/row-level access. Macie for PII discovery. Glue Catalog for metadata. Lineage via Glue/Datahub. Retention policies by jurisdiction (7 years for financial). Object Lock for audit. Crypto-shredding for GDPR erasure. Access reviews quarterly. **Q910 — Design a migration from on-prem Oracle to Snowflake.** **A910** — Inventory objects, profile data, map types (NUMBER → NUMBER, DATE → DATE, CLOB → VARCHAR). Use Snowflake's SnowConvert or manual rewrite for PL/SQL. Stage data in S3, load via COPY with VALIDATION_MODE. Reconcile counts, sums, row hashes per table. Parallel run old and new for 2-3 cycles. Cut over with rollback plan. Validate downstream reports and interfaces. **Q911 — Design a CDC pipeline from PostgreSQL to S3.** **A911** — Use Debezium (Kafka Connect) or AWS DMS for CDC. Publish changes to MSK/Kinesis. Consume with Spark Streaming or Lambda. Write to S3 Bronze partitioned by date and operation (I/U/D). Merge into Silver with Delta/Iceberg MERGE. Handle schema evolution via registry. Test: insert, update, delete, late-arriving, duplicate. Validate counts and hashes. **Q912 — Design a data quality framework for 100 pipelines.** **A912** — Central metadata store (YAML per dataset) with rules, thresholds, severity. Shared rule library (Python/PySpark). Runner orchestrates checks post-load. Results in a Delta table. Dashboard in Power BI. Critical failures block downstream. Onboarding: template + checklist. Alerting via PagerDuty/Slack. Covered in Section 7.3 of the prep kit. **Q913 — Design a multi-region DR for a data platform.** **A913** — Active-passive: primary region handles all traffic; standby replicates data (S3 CRR, MSK replication, RDS read replica) and is promoted on failure. RPO < 5 min, RTO < 1 hour. DNS failover via Route 53. Quarterly game days. Validate data freshness and downstream consumers after failover. Document runbooks. **Q914 — Design a cost-optimized Spark platform.** **A914** — Auto-scaling clusters (EMR/Glue/Databricks), spot instances for non-critical jobs, right-sized executors, partition pruning, file compaction, AQE, avoid UDFs, cache reused DataFrames, use columnar formats, schedule off-peak, monitor cost per job, budget alarms. **Q915 — Design a data contract enforcement system.** **A915** — Producer registers schema in a registry (Avro/Protobuf) or a contract file (YAML). CI/CD gate compares new schema with registered version; fails on breaking changes. Consumer tests against the contract. Runtime validation at ingestion. Alerts on schema drift. Versioned contracts with deprecation policy. --- ## SECTION 59 — DATA ENGINEERING CODING CHALLENGES **Q916 — Given a list of trades, find the top 3 accounts by total amount.** **A916** ```python from collections import defaultdict totals = defaultdict(float) for t in trades: totals[t['account_id']] += t['amount'] top3 = sorted(totals.items(), key=lambda x: -x[1])[:3] ``` **Q917 — Given two lists, find the intersection without using sets.** **A917** ```python def intersect(a, b): b_set = set(b) return [x for x in a if x in b_set] ``` **Q918 — Given a string, find the longest substring without repeating characters.** **A918** ```python def longest_unique(s): seen = {} start = max_len = 0 for i, ch in enumerate(s): if ch in seen and seen[ch] >= start: start = seen[ch] + 1 seen[ch] = i max_len = max(max_len, i - start + 1) return max_len ``` **Q919 — Given a list of intervals, merge overlapping intervals.** **A919** ```python def merge(intervals): intervals.sort(key=lambda x: x[0]) out = [] for start, end in intervals: if out and start <= out[-1][1]: out[-1][1] = max(out[-1][1], end) else: out.append([start, end]) return out ``` **Q920 — Given a stream of numbers, find the median at any point.** **A920** — Use two heaps: max-heap for the lower half, min-heap for the upper half. Balance sizes; median is the top of the larger heap or the average of both tops. O(log n) insert, O(1) median. **Q921 — Given a table of orders, find customers who ordered every month for 6 consecutive months.** **A921** — Use window functions: for each customer, order by month, compute `month - row_number()` to group consecutive months, then `HAVING COUNT(*) >= 6`. **Q922 — Given a table of page views, find the top 5 pages by unique viewers per day.** **A922** — `SELECT date, page, COUNT(DISTINCT user_id) AS viewers FROM views GROUP BY date, page QUALIFY ROW_NUMBER() OVER (PARTITION BY date ORDER BY viewers DESC) <= 5`. **Q923 — Given two tables, find rows that exist in A but not B and vice versa.** **A923** — `SELECT * FROM A EXCEPT SELECT * FROM B` and the reverse. Or `FULL OUTER JOIN` with a WHERE on NULL. Use EXCEPT ALL if duplicates matter. **Q924 — Given a table with start and end times, find overlapping sessions per user.** **A924** — Self-join on user_id and overlapping ranges: `a.start < b.end AND b.start < a.end AND a.id < b.id`. Filter to distinct pairs. Or use a sweep-line algorithm in Python. **Q925 — Given a large file, find the most frequent word without loading it all.** **A925** — Stream line by line, use a Counter. For truly huge files, use a distributed approach (Spark word count) or external sort + count. **Q926 — Given a table of transactions, detect duplicate charges within 5 minutes.** **A926** — Self-join on account_id and amount where timestamps differ by < 5 minutes. Or use `LAG` window function to compare with the previous transaction. **Q927 — Given a list of employees with managers, find the depth of the hierarchy.** **A927** — Recursive CTE: start from the top (manager_id IS NULL), increment level, join employees on manager_id = previous level's emp_id. Max level is the depth. **Q928 — Given a table of stock prices, find the max drawdown per security.** **A928** — Window function: running max of price; drawdown = (price - running_max) / running_max; min drawdown per security. **Q929 — Given a table of events, find sessions (gaps > 30 minutes start a new session).** **A929** — Use `LAG` to get the previous event time; flag a new session when gap > 30 min; `SUM` the flag over time to get a session ID per user. **Q930 — Given a table of orders, find the second order date per customer.** **A930** — `ROW_NUMBER() OVER (PARTITION BY customer_id ORDER BY order_date)` = 2. Use `DENSE_RANK` if multiple orders per day count as one. --- ## SECTION 60 — CAREER, COMMUNICATION & NEGOTIATION **Q931 — How do you negotiate salary?** **A931** — (1) Research market rates (Glassdoor, Levels.fyi, peers). (2) Know your walk-away number. (3) Let them make the first offer if possible. (4) Negotiate the total package (base, bonus, equity, benefits, remote). (5) Be polite and firm. (6) Get it in writing. (7) Never lie about current comp — it can be verified. **Q932 — How do you handle a lowball offer?** **A932** — Thank them, express enthusiasm, then present your market data and your value. Ask if they can do better on base, signing bonus, or equity. If they can't, decide if the role/company is worth the gap. Don't accept out of desperation. **Q933 — How do you explain a career gap?** **A933** — Be honest and brief. Focus on what you did during the gap: learning, lab work, certifications, family, health. Then pivot to why you're excited about this role. Don't over-apologize. **Q934 — How do you handle a question about why you're leaving?** **A934** — Stay positive. Focus on what you're moving toward, not what you're running from. "I've grown a lot, and I'm looking for a role with a stronger focus on data quality and platform reliability." Never badmouth your current employer. **Q935 — How do you handle a question about a former manager you didn't like?** **A935** — Stay professional. Focus on the situation, not the person. "We had different working styles, and I learned to communicate more explicitly about expectations." Never blame. **Q936 — How do you handle a question about your biggest failure?** **A936** — Pick a real, small failure. Use STAR. Focus on what you learned and what you changed. "I missed a duplicate check; duplicates reached Gold. I fixed the cause, reran idempotently, and added a permanent uniqueness check." **Q937 — How do you handle a question about a time you disagreed with a decision?** **A937** — Show that you can disagree respectfully and commit. "I raised my concern with data, the team decided differently, and I supported the decision. Later we revisited and adjusted." **Q938 — How do you handle a question about a time you had to learn something quickly?** **A938** — Use STAR. "We needed Databricks; I'd only used on-prem Spark. I built a small end-to-end pipeline in a lab, read the docs, asked targeted questions, and delivered on time with full reconciliation." **Q939 — How do you handle a question about a time you mentored someone?** **A939** — Use STAR. "A junior engineer was new to PySpark. I pair-programmed on a validation job, reviewed their code line by line, and explained the 'why' behind each check. They independently built the next job with minimal review." **Q940 — How do you handle a question about a time you took ownership beyond your role?** **A940** — Use STAR. "A DQ issue was found in a report after my testing was signed off. I traced it to a join in the upstream pipeline, raised it with the upstream team, and proposed an automated check. The check was added and the issue never recurred." **Q941 — How do you handle a question about a time you missed a deadline?** **A941** — Use STAR. "A test cycle was delayed by a late dependency. I prioritized risk-based testing, communicated daily, and escalated early with a revised plan. Critical paths were tested on time; non-critical deferred with sign-off." **Q942 — How do you handle a question about working with a difficult stakeholder?** **A942** — Focus on understanding their priorities, communicating clearly, and finding common ground. Use evidence, not emotion. Escalate only when necessary. **Q943 — How do you handle a question about remote work?** **A943** — Emphasize: clear communication, async updates, documented decisions, overlap hours for collaboration, and results over hours. Mention tools: Slack, Teams, Confluence, Jira, Git. **Q944 — How do you handle a question about relocation?** **A944** — Be honest about your constraints. If you can relocate, say so. If not, ask about remote or hybrid options. Don't lie — it will backfire. **Q945 — How do you handle a question about notice period?** **A945** — State it clearly (typically 30–90 days in India). Ask if they can accommodate. If not, discuss options (buyout, early release). Don't commit to something you can't deliver. **Q946 — What should you do after receiving an offer?** **A946** — (1) Thank them. (2) Ask for the written offer. (3) Review terms: base, bonus, equity, benefits, start date, location. (4) Negotiate if needed. (5) Accept or decline politely. (6) Stay in touch even if declining — the industry is small. **Q947 — How do you evaluate a job offer beyond salary?** **A947** — (1) Role: growth, learning, ownership. (2) Team: manager, peers, culture. (3) Company: stability, mission, stage. (4) Tech: stack, scale, modernization. (5) Work-life: hours, remote, on-call. (6) Total comp: base, bonus, equity, benefits. (7) Location and commute. **Q948 — How do you build a personal brand as a data engineer?** **A948** — (1) Write about what you learn (blog, LinkedIn). (2) Contribute to open source (dbt, Airflow, Great Expectations). (3) Speak at meetups or conferences. (4) Mentor juniors. (5) Build a portfolio (GitHub) of pipelines and frameworks. (6) Network genuinely. **Q949 — How do you stay employable in a changing market?** **A949** — (1) Master fundamentals (SQL, Python, distributed systems). (2) Learn one cloud deeply and one adjacent. (3) Follow trends (lakehouse, streaming, AI). (4) Build transferable skills (system design, DQ, governance). (5) Keep a lab. (6) Network. (7) Certifications help but don't replace experience. **Q950 — What is the future of the data engineer role?** **A950** — Trends: (1) Convergence with analytics engineering (dbt, SQL-first). (2) More platform engineering (IaC, observability). (3) AI/ML integration (feature stores, vector DBs). (4) Governance and privacy by design. (5) Streaming-first. (6) Serverless and managed services. (7) Cost optimization (FinOps). The role is broadening, not disappearing. --- ## FINAL COUNT | Section | Topic | Q&As | |---|---|---| | 1–21 | Original bank | 445 | | 22–35 | First gap additions | 185 | | 36–48 | Second gap additions | 170 | | 49–60 | Third gap additions | 150 | | **Total** | | **950** | --- ## WHAT WAS STILL MISSING — SUMMARY | Gap Area | Why It Matters | New Section | |---|---|---| | Hadoop ecosystem (HDFS, YARN, MapReduce, Hive, HBase) | On your CV lab — must defend | 49 | | Apache Iceberg deep dive | On your CV lab — must defend | 50 | | Apache Flink | On your CV lab — must defend | 51 | | Trino/Presto | On your CV lab — must defend | 52 | | Prometheus/Grafana | On your CV lab — must defend | 53 | | Advanced Python (decorators, generators, typing, async, GIL) | Senior Python questions | 54 | | Advanced pytest (fixtures, mock, parametrize, coverage) | On your CV — must defend | 55 | | Data quality frameworks comparison | Common design question | 56 | | Data engineering anti-patterns | Senior-level maturity | 57 | | System design framework | Senior-level interviews | 58 | | Coding challenges | Live coding rounds | 59 | | Career, communication & negotiation | End-to-end interview | 60 | --- **Total Q&A bank now: 950 questions.** This covers every line on your CV, every tool in your lab, every domain in both architecture diagrams, every failure area, every common interview pattern, coding challenges, system design, and career skills. Use it as a complete reference; prioritize by the job description and your own hands-on experience. --- Source file: Pasted text (5).txt # Still Missing — Final Q&A Bank (Q951–Q1,100) You are right to keep asking. After 950 questions, the bank is comprehensive on **technical depth**. But several **high-value areas are still missing** — leadership, GenAI data pipelines, non-banking domains, interview red flags, take-home assignments, whiteboard tactics, first-90-days, cross-functional work, metrics, ethics, and a mock interview transcript. Below are **150 more Q&As** in these areas. After this, the bank is genuinely complete for ETL/Data Engineer roles at XXX, TCS, Accenture, Amazon, Microsoft, and product firms. Further additions would be marginal. --- ## SECTION 61 — LEADERSHIP & MANAGEMENT IN DATA ENGINEERING **Q951 — What is the difference between an IC and a lead data engineer?** **A951** — IC: owns technical delivery, writes code, reviews others. Lead: owns the technical direction, unblocks the team, mentors, communicates with stakeholders, and still reviews/designs. The lead is accountable for outcomes, not just tasks. **Q952 — How do you run a code review?** **A952** — (1) Understand the intent first. (2) Check correctness: logic, edge cases, nulls, duplicates. (3) Check performance: joins, partitioning, shuffle. (4) Check security: secrets, PII, access. (5) Check tests: coverage, edge cases. (6) Check readability: naming, comments. (7) Be specific and kind. (8) Approve only when you would run it in production. **Q953 — How do you mentor a junior data engineer?** **A953** — (1) Pair on the first task. (2) Explain the "why" behind each design choice. (3) Review their code line by line early, then step back. (4) Give them a small end-to-end task with clear success criteria. (5) Encourage them to write the DQ checks and run the reconciliation. (6) Debrief after each task — what went well, what to improve. **Q954 — How do you run a sprint for a data team?** **A954** — (1) Refinement: clarify stories, estimate, identify dependencies. (2) Planning: commit to sprint scope. (3) Daily standup: blockers, progress, coordination. (4) Mid-sprint check: adjust scope if needed. (5) Demo: show working data products. (6) Retrospective: what to improve. Protect the team from scope creep. **Q955 — How do you handle an underperforming team member?** **A955** — (1) Identify specific gaps with examples. (2) Have a private, direct conversation. (3) Agree on a plan with clear expectations and timeline. (4) Provide support: pairing, training, clearer tasks. (5) Follow up regularly. (6) Escalate to HR if no improvement. Focus on behavior and outcomes, not personality. **Q956 — How do you prioritize a backlog?** **A956** — (1) Business impact: revenue, compliance, risk. (2) Urgency: deadlines, regulatory. (3) Effort: quick wins vs long projects. (4) Dependencies: what unblocks others. (5) Technical debt: reliability, performance. (6) Stakeholder alignment. Use a scoring framework (RICE, WSJF) but apply judgment. **Q957 — How do you handle a stakeholder who keeps changing requirements?** **A957** — (1) Understand the underlying need, not just the request. (2) Document the change and its impact (scope, time, cost). (3) Present options: defer, swap, or expand. (4) Get written agreement. (5) Set a change freeze before release. (6) Escalate if it becomes a pattern. **Q958 — How do you hire a data engineer?** **A958** — (1) Define the role: IC vs lead, batch vs streaming, cloud, domain. (2) Screen for fundamentals: SQL, Python, one cloud. (3) Technical interview: coding, SQL, design, project deep-dive. (4) Behavioral: ownership, collaboration, learning. (5) Check references. (6) Assess culture fit and growth potential. **Q959 — What is the difference between a data engineer and an analytics engineer?** **A959** — Data engineer: builds pipelines, ingestion, storage, orchestration, reliability. Analytics engineer: builds transformation models (dbt), semantic layers, and data products for analysts. Overlap at SQL and modeling. Many teams combine the roles. **Q960 — How do you build a data team from scratch?** **A960** — (1) Hire a senior engineer first (sets standards). (2) Add an analytics engineer (data products). (3) Add a platform engineer (infra, IaC). (4) Add a junior for capacity. (5) Define ownership: domains, pipelines, data products. (6) Establish standards: code review, testing, DQ, documentation. (7) Invest in tooling: catalog, lineage, observability. --- ## SECTION 62 — GENAI / AI DATA PIPELINES (VERY CURRENT) **Q961 — What is a vector database?** **A961** — A database optimized for storing and querying high-dimensional vectors (embeddings). Supports similarity search (cosine, dot product, Euclidean). Examples: Pinecone, Weaviate, Milvus, pgvector, Chroma. Used in RAG, semantic search, recommendations. **Q962 — What is RAG (Retrieval-Augmented Generation)?** **A962** — A pattern where an LLM retrieves relevant documents from a vector store and uses them as context to generate an answer. Reduces hallucination and grounds responses in your data. Pipeline: chunk documents → embed → store in vector DB → retrieve top-k → pass to LLM. **Q963 — How do you build a data pipeline for embeddings?** **A963** — (1) Ingest source documents (PDFs, HTML, DB rows). (2) Chunk into passages (with overlap). (3) Clean and normalize text. (4) Call an embedding model (OpenAI, Cohere, open-source). (5) Store vectors in a vector DB with metadata. (6) Incremental updates: only new/changed docs. (7) Validate: no empty chunks, no PII leakage, embedding dimension matches. **Q964 — What is a feature store for ML?** **A964** — A centralized repository for ML features with offline (training) and online (serving) stores. Ensures consistency between training and serving. Examples: Feast, Tecton, Databricks Feature Store. Prevents training-serving skew. **Q965 — What is model monitoring?** **A965** — Tracking model performance and data drift in production. Metrics: prediction distribution, feature distribution, ground-truth accuracy (when labels arrive), latency, business KPIs. Alerts on drift. Retrain when drift exceeds a threshold. **Q966 — How do you build a data pipeline for LLM fine-tuning?** **A966** — (1) Collect domain data (documents, Q&A pairs). (2) Clean and deduplicate. (3) Format as instruction/response pairs. (4) Split train/validation. (5) Version datasets (DVC, LakeFS). (6) Track experiments (MLflow, W&B). (7) Evaluate on held-out set. (8) Deploy and monitor. PII handling is critical. **Q967 — What is a data flywheel?** **A967** — A loop where production usage generates data that improves the model, which improves the product, which generates more usage. Requires pipelines to capture user feedback, label it, and retrain. Common in recommendation and search systems. **Q968 — How do you handle PII in AI pipelines?** **A968** — (1) Detect PII at ingestion (Macie, Presidio, regex). (2) Mask or tokenize before embedding. (3) Restrict access to raw data. (4) Audit usage. (5) Comply with data residency and retention. (6) Never send sensitive data to unapproved external APIs. **Q969 — What is prompt engineering from a data perspective?** **A969** — Structuring the input to an LLM for reliable output. From a data view: version prompts, A/B test them, log inputs/outputs, measure quality, and treat prompts as code (review, test, deploy). Store prompt-response pairs for evaluation and fine-tuning. **Q970 — What is a data contract for AI features?** **A970** — A versioned agreement on feature inputs and outputs: schema, semantics, freshness, quality thresholds, and ownership. Critical when features are shared between training and serving. Enforced via schema registry and DQ checks. --- ## SECTION 63 — DOMAIN-SPECIFIC BEYOND BANKING **Q971 — What are common data challenges in healthcare?** **A971** — (1) HIPAA compliance: PHI must be encrypted, access-audited, retention-controlled. (2) HL7/FHIR standards for clinical data. (3) ICD codes for diagnosis. (4) Claims vs clinical data reconciliation. (5) Patient identity matching (MPI). (6) Time-sensitive data (medication, lab results). (7) Interoperability across EHRs. **Q972 — What are common data challenges in retail / e-commerce?** **A972** — (1) High-volume clickstream and transaction data. (2) Session stitching across devices. (3) Inventory reconciliation across channels. (4) Pricing and promotion logic. (5) Customer 360 (identity resolution). (6) Real-time personalization. (7) Seasonal spikes (Black Friday). **Q973 — What are common data challenges in telecom?** **A973** — (1) CDR (call detail records) at scale. (2) Network performance data (5G, cell towers). (3) Churn prediction. (4) Billing reconciliation. (5) Usage metering. (6) Roaming data. (7) Subscriber identity across services. You have direct experience from your telecom churn project. **Q974 — What are common data challenges in insurance?** **A974** — (1) Policy and claims data. (2) Actuarial models. (3) Fraud detection. (4) Regulatory reporting (Solvency II, IFRS 17). (5) Claims lifecycle (FNOL → settlement). (6) Risk pricing. (7) Customer 360. **Q975 — What are common data challenges in manufacturing / IoT?** **A975** — (1) Sensor data at high frequency. (2) Time-series storage and query. (3) Edge-to-cloud pipelines. (4) Anomaly detection. (5) Predictive maintenance. (6) Supply chain visibility. (7) Data quality from unreliable sensors. **Q976 — What are common data challenges in media / streaming?** **A976** — (1) Viewing events at scale. (2) Content metadata. (3) Recommendation features. (4) Ad attribution. (5) Rights management. (6) Real-time personalization. (7) Sessionization across devices. **Q977 — What are common data challenges in supply chain / logistics?** **A977** — (1) Shipment tracking events. (2) Inventory visibility. (3) Demand forecasting. (4) Supplier data integration. (5) Route optimization. (6) Exception handling (delays, damage). (7) Cross-border compliance. **Q978 — What is a data product for a non-banking domain?** **A978** — A curated dataset with an owner, SLA, schema, DQ contract, and consumers. Examples: "Customer 360" in retail, "Patient timeline" in healthcare, "Network performance index" in telecom. Treated as a product, not a pipeline byproduct. --- ## SECTION 64 — INTERVIEW RED FLAGS & DEAL-BREAKERS **Q979 — What are common red flags in a data engineering interview?** **A979** — (1) Bluffing about tools you haven't used. (2) Not asking clarifying questions. (3) Claiming credit for team work. (4) Badmouthing former employers. (5) Not knowing your own CV. (6) Ignoring edge cases (nulls, duplicates, timezones). (7) Not mentioning testing or DQ. (8) Not discussing trade-offs. (9) Monologuing. (10) No questions at the end. **Q980 — What should you never say in an interview?** **A980** — (1) "I don't know" without trying. (2) "That's not my job." (3) "I've never made a mistake." (4) "My last team was terrible." (5) "I just followed the spec." (6) "I don't test my code." (7) "I don't care about the business." (8) Any lie about experience. (9) "I'll figure it out later" for a design decision. (10) "I only work on X." **Q981 — What are signs the interviewer is losing interest?** **A981** — (1) Short responses. (2) Looking at the clock. (3) Interrupting. (4) Repeating the question. (5) Not probing deeper. Recovery: be concise, return to the structure (symptom → scope → proof → fix), and ask if they want more depth. **Q982 — How do you recover from a bad answer?** **A982** — (1) Don't panic. (2) Acknowledge: "Let me reconsider." (3) Restate the problem. (4) Give a structured approach, even if incomplete. (5) Say what you would look up. (6) Move on. Interviewers remember how you recover, not the initial miss. **Q983 — How do you handle an interviewer who is hostile or dismissive?** **A983** — Stay calm, professional, and evidence-based. Answer the question, ask for clarification if needed. Don't get defensive. If it persists, it's a signal about the culture — you may not want the role. **Q984 — What if you don't know a tool the JD requires?** **A984** — Use the gap template: "I haven't used [tool] in a project. The validation approach is the same: counts, aggregates, row-level diff, business rules. In [tool] I would use [equivalent feature]. I have done the same in [tool you know], and I ramp up quickly." **Q985 — What if the interviewer asks about a project you didn't work on?** **A985** — Be honest: "I didn't work on that project, but based on my experience with [similar project], I would approach it by [method]." Never claim credit you don't deserve. **Q986 — How do you handle a question you've never heard before?** **A986** — (1) Repeat it to confirm understanding. (2) Break it into parts. (3) Solve the parts you know. (4) State assumptions. (5) Think aloud. (6) Ask for a hint if stuck. (7) Never go silent. **Q987 — What is the biggest mistake candidates make in a data engineering interview?** **A987** — Focusing only on tools, not on the reasoning. Interviewers want to hear *why* you chose an approach, *what* could go wrong, and *how* you would validate. The rhythm (what I saw → where I narrowed it down → proof → fix → prevention) matters more than naming services. --- ## SECTION 65 — TAKE-HOME ASSIGNMENTS & PAIR PROGRAMMING **Q988 — What is a typical data engineering take-home?** **A988** — (1) Build a small ETL pipeline from a CSV/API to a database. (2) Write SQL for a set of analytical questions. (3) Design a schema for a given use case. (4) Debug a broken pipeline. (5) Write tests for a transformation. Time: 2–8 hours. Deliverable: code + README + tests. **Q989 — How do you approach a take-home?** **A989** — (1) Read the requirements twice. (2) Clarify ambiguities in writing. (3) Design before coding. (4) Write tests first. (5) Keep it simple and working. (6) Document assumptions, trade-offs, and what you'd do with more time. (7) Submit on time. (8) Be ready to walk through and extend it live. **Q990 — What do interviewers look for in a take-home?** **A990** — (1) Correctness: does it work? (2) Testing: are there tests? (3) Structure: is it readable and modular? (4) Edge cases: nulls, duplicates, empty input. (5) Documentation: README, assumptions. (6) Judgment: did you over-engineer or under-deliver? (7) Honesty: what you didn't do. **Q991 — How do you handle a take-home that's too large?** **A991** — Do a subset well, document what you skipped, and explain why. Add a "what I would do with more time" section. Don't submit something broken. Quality over quantity. **Q992 — What is a pair programming interview?** **A992** — You and the interviewer solve a problem together in real time. They assess collaboration, communication, and problem-solving — not just the final answer. You may drive or navigate. **Q993 — How do you succeed in a pair programming interview?** **A993** — (1) Think aloud. (2) Ask questions. (3) Accept feedback. (4) Don't dominate. (5) Suggest a plan before coding. (6) Write clean code. (7) Test with examples. (8) Say "I'd look this up" for API details. (9) Be pleasant. **Q994 — What if you get stuck in a pair programming round?** **A994** — (1) Say what you know. (2) Solve a simpler version. (3) Ask a targeted question. (4) Propose a brute-force first, then optimize. (5) Never sit in silence. **Q995 — How do you handle a take-home you suspect was designed to exploit free labor?** **A995** — If the scope is unreasonable for an interview (production pipeline, weeks of work), ask for clarification. If they insist, decline politely and explain. Legitimate take-homes are bounded (2–8 hours) and clearly for evaluation. --- ## SECTION 66 — WHITEBOARD SYSTEM DESIGN TACTICS **Q996 — How do you structure a whiteboard data system design answer?** **A996** — (1) Clarify: sources, volume, latency, SLA, consumers, compliance. (2) Sketch: ingestion → storage → processing → serving. (3) Annotate: technologies, partitioning, format. (4) Discuss: reliability (idempotency, retries, DQ), scale (parallelism, cost), operations (monitoring, alerting). (5) Trade-offs: alternatives and why you chose this. (6) Failure modes: what could go wrong and how you'd detect it. **Q997 — What should you always include in a data system design?** **A997** — (1) Idempotency. (2) DQ gates. (3) Quarantine / DLQ. (4) Monitoring and alerting. (5) Backfill and replay. (6) Security and governance. (7) Cost. (8) Failure modes. Without these, the design is incomplete. **Q998 — How do you handle a design question at a scale you haven't worked with?** **A998** — Be honest about your experience, then reason from first principles: partition by date, parallelize, use columnar formats, cache, broadcast small dimensions. Use numbers: 1 TB/day = ~12 MB/s average. Ask about peak vs average. Explain what you'd test. **Q999 — How do you compare two design options on a whiteboard?** **A999** — (1) State the criteria: latency, cost, complexity, reliability, maintainability. (2) Score each option. (3) Pick one and justify. (4) Mention what would change your mind. Interviewers value structured trade-off reasoning over a "correct" answer. **Q1000 — What if the interviewer disagrees with your design?** **A1000** — Listen, ask what they'd do differently, and adapt. Don't defend blindly. Say: "That's a good point — if [constraint], then [alternative] would be better." Show you can incorporate feedback. **Q1001 — How do you end a design answer?** **A1001** — Summarize: the architecture, the key trade-offs, the failure modes, and how you'd validate. Then ask: "Would you like me to go deeper on any part?" This signals completeness and invites follow-ups. --- ## SECTION 67 — FIRST 90 DAYS ON THE JOB **Q1002 — What should you do in your first week as a data engineer?** **A1002** — (1) Understand the business: what data products, who consumes them. (2) Map the architecture: sources, layers, orchestration, serving. (3) Meet the team and key stakeholders. (4) Read the docs: runbooks, mappings, DQ rules. (5) Run a pipeline end-to-end. (6) Ask questions — you're expected to. **Q1003 — What should you do in your first 30 days?** **A1003** — (1) Ship a small, low-risk change. (2) Add a DQ check or alert. (3) Document one pipeline. (4) Identify one pain point and propose a fix. (5) Build relationships with upstream and downstream teams. **Q1004 — What should you do in your first 90 days?** **A1004** — (1) Own a pipeline or data product end-to-end. (2) Improve reliability: reduce failures, add monitoring. (3) Improve cost: identify waste. (4) Mentor or onboard a new hire. (5) Propose a roadmap item. (6) Earn trust through consistent delivery. **Q1005 — How do you onboard into a large, undocumented data platform?** **A1005** — (1) Profile the data: what's in each layer, how fresh, who owns it. (2) Trace lineage: pick a report and follow it back to source. (3) Read the code: jobs, configs, DQ rules. (4) Interview the team. (5) Write the docs you wish existed. (6) Validate by reproducing a known number. **Q1006 — How do you handle being the only data engineer on a team?** **A1006** — (1) Prioritize by business impact. (2) Automate aggressively. (3) Document everything. (4) Build self-service for common asks. (5) Set SLAs and manage expectations. (6) Escalate for headcount with evidence (incidents, backlog). **Q1007 — How do you handle technical debt in a legacy pipeline?** **A1007** — (1) Quantify: incidents, cost, maintenance hours. (2) Propose incremental modernization. (3) Build the new alongside the old (parallel run). (4) Validate equivalence (counts, sums, hashes). (5) Cut over with rollback. (6) Never big-bang rewrite without a rollback. --- ## SECTION 68 — CROSS-FUNCTIONAL COLLABORATION **Q1008 — How do you work with data scientists?** **A1008** — (1) Provide clean, documented, versioned datasets. (2) Agree on feature contracts. (3) Support experimentation with fast iteration. (4) Help move models to production (feature stores, batch scoring). (5) Monitor for drift. (6) Document lineage for reproducibility. **Q1009 — How do you work with data analysts?** **A1009** — (1) Provide trusted Gold tables with clear definitions. (2) Document metrics and grain. (3) Enable self-service with a semantic layer. (4) Support ad-hoc queries with governed data. (5) Train them on tools. (6) Solicit feedback on data quality. **Q1010 — How do you work with product managers?** **A1010** — (1) Translate business needs into data requirements. (2) Communicate trade-offs (scope, time, quality). (3) Provide SLAs for data freshness and quality. (4) Report on delivery and incidents. (5) Align on priorities. **Q1011 — How do you work with software engineers?** **A1011** — (1) Agree on data contracts for events and APIs. (2) Review schema changes before deployment. (3) Coordinate on release timing. (4) Provide feedback on data-producing code. (5) Share observability (logs, metrics). **Q1012 — How do you work with DevOps/SRE?** **A1012** — (1) Use the same CI/CD pipelines. (2) Share IaC standards. (3) Integrate monitoring and alerting. (4) Coordinate on incident response. (5) Follow security and access policies. **Q1013 — How do you work with compliance/legal?** **A1013** — (1) Classify data by sensitivity. (2) Implement retention and residency rules. (3) Provide audit trails. (4) Support right-to-erasure requests. (5) Document lineage for regulatory reporting. **Q1014 — How do you work with external vendors?** **A1014** — (1) Agree on data contracts and SLAs. (2) Validate vendor data on arrival (schema, count, checksum, quality). (3) Monitor for drift. (4) Reconcile against internal sources. (5) Escalate issues with evidence. --- ## SECTION 69 — DATA ENGINEERING METRICS & KPIs **Q1015 — What KPIs is a data engineer measured on?** **A1015** — (1) Pipeline reliability: uptime, failure rate, MTTR. (2) Freshness: data available by SLA. (3) Data quality: DQ pass rate, defects, escaped defects. (4) Cost: cost per job, per TB, per pipeline. (5) Delivery: stories completed, on-time delivery. (6) Stakeholder satisfaction. **Q1016 — What is MTTR?** **A1016** — Mean Time To Recovery: the average time to restore service after an incident. Lower is better. Tracked per pipeline or platform-wide. Improved by monitoring, runbooks, and automation. **Q1017 — What is data freshness?** **A1017** — How recently the dataset reflects the source/business event. Measured as the lag between event time and availability in the target. Monitored against an SLA (e.g., Gold freshness < 5 minutes). **Q1018 — What is a data quality scorecard?** **A1018** — A dashboard showing DQ metrics per dataset: completeness %, validity %, uniqueness %, freshness, reconciliation pass rate, defect count. Used to track trends and prioritize remediation. **Q1019 — What is cost per query or cost per job?** **A1019** — The cloud cost attributable to a specific query or job. Tracked via tags, CloudWatch, or cost allocation tools. Used to identify waste, optimize, and charge back to teams. **Q1020 — What is pipeline utilization?** **A1020** — The percentage of time a pipeline or cluster is doing useful work vs idle. Low utilization = waste. High utilization = potential bottleneck. Used for capacity planning. **Q1021 — What is defect escape rate?** **A1021** — The percentage of defects found in production vs during testing. Lower is better. Improved by better test coverage, DQ gates, and shift-left testing. **Q1022 — What is the difference between an incident and a problem?** **A1022** — Incident: a single event (a pipeline failed). Problem: the underlying cause of one or more incidents (a fragile join that fails on schema change). Fix incidents fast; fix problems permanently. --- ## SECTION 70 — CONTRACT VS FULL-TIME & GLOBAL MOBILITY **Q1023 — What is the difference between full-time and contract data engineering?** **A1023** — Full-time: salary, benefits, long-term, career growth, internal mobility. Contract: higher hourly rate, no benefits, shorter term, less stability, often via a vendor. Contract suits specialists; full-time suits career builders. **Q1024 — What should you check before accepting a contract role?** **A1024** — (1) Rate: compare to full-time equivalent (include benefits). (2) Duration and renewal terms. (3) Notice period for early termination. (4) Tax and compliance. (5) IP and non-compete clauses. (6) Payment terms. (7) Whether the client can hire you full-time later. **Q1025 — How do you handle a notice period buyout?** **A1025** — (1) Check your contract for the buyout amount. (2) Ask the new employer if they'll cover it. (3) Negotiate a shorter notice with your current manager. (4) Get everything in writing. (5) Plan a clean handover. **Q1026 — What is H1B and how does it affect data engineers?** **A1026** — H1B is a US work visa for specialty occupations. Requires employer sponsorship, lottery (cap-subject), and prevailing wage. Alternatives: L1 (intra-company transfer), O1 (extraordinary ability), remote work from India, or relocate to Canada/EU. **Q1027 — What is an EU Blue Card?** **A1027** — A work permit for highly skilled non-EU workers. Requires a job offer, recognized qualification, and salary threshold. Leads to permanent residency. Common in Germany, Netherlands, and other EU countries. **Q1028 — How do you evaluate relocation for a data engineering role?** **A1028** — (1) Cost of living vs salary. (2) Visa and long-term residency. (3) Family considerations. (4) Career growth in that market. (5) Tax implications. (6) Culture and language. (7) Return options. **Q1029 — What is the difference between a vendor and a product company interview?** **A1029** — Vendor (XXX, TCS): project-based, client-facing, broader stack, communication matters. Product (Amazon, Google): deep technical, ownership, system design, scale. Product interviews are harder on design; vendor interviews are broader on tools. --- ## SECTION 71 — MOCK INTERVIEW TRANSCRIPT (ROLE-PLAY) **Q1030 — Mock: "Walk me through your XXX project."** **A1030** — "I built PySpark, Python and SQL pipelines that ingest investment data from Azure Data Lake and AWS S3. The data includes trades, positions, prices, FX rates, and security reference data. I owned the validation layer: schema validation, null checks, business-rule checks, and source-to-target reconciliation, so NAV, risk, and fund-performance numbers could be trusted. I also tuned the SQL Server queries behind the KPIs and supported the legacy-to-Azure migration." **Q1031 — Mock: "What was the hardest defect you found?"** **A1031** — "After a release, reconciliation showed target market value higher than source for some funds. I compared counts and sums layer by layer. Raw matched, curated did not, and the jump came right after the join to the security reference table. A duplicate check on the reference key showed several effective-dated rows per security, so each position was multiplied. I joined to the latest effective row using ROW_NUMBER, reran the affected dates idempotently, and added two permanent checks: row count must not change across the join, and the reference key must be unique." **Q1032 — Mock: "Write SQL to find duplicate trade IDs."** **A1032** — "```sql SELECT trade_id, COUNT(*) AS cnt FROM tgt.trades GROUP BY trade_id HAVING COUNT(*) > 1; ```" Follow-up: "If the business key is composite, I include all key columns in the GROUP BY." **Q1033 — Mock: "How do you handle a skewed join in PySpark?"** **A1033** — "First I confirm skew in the Spark UI — one or a few tasks run much longer. Then I profile key distribution. If one key dominates, I use salting: add a random suffix 0..N-1 to the hot key on the big side, replicate the small side N times with each suffix, and join on key plus salt. If the dimension is small enough, I broadcast it instead. I also enable AQE skew-join handling." **Q1034 — Mock: "A pipeline is green but Gold has no data. What do you do?"** **A1034** — "First, quantify: which date, which dataset, how big. Then check inputs: file present, size above zero, row count vs trailer. Then check the run: parameters, logs, first error. Then check the target: partition written, freshness. In a past case, the upstream file arrived late and the job read zero rows without error. I added pre-checks (file exists, row count above zero), fail-fast with an alert, a wait-and-retry until the SLA cut-off, and a freshness check on Gold." **Q1035 — Mock: "How would you design a daily reporting platform for 1 TB/day?"** **A1035** — "Sources: RDBMS, files, APIs. Ingest to S3 Bronze partitioned by date/source. Orchestrate with Step Functions or Airflow. Transform with Glue/Spark into Silver (cleansed, conformed) and Gold (business aggregates). DQ gates with Great Expectations. Serve via Athena/Redshift/QuickSight. SLA: 06:00 IST freshness. Idempotent daily partitions. Backfill support. Cost: auto-scaling clusters, S3 lifecycle to Glacier after 90 days. Failure modes: late files, schema drift, skew, DQ failures — each with a detection signal and a recovery action." **Q1036 — Mock: "Why are you leaving your current role?"** **A1036** — "I've grown a lot in my current role, and I'm looking for a role that combines data engineering with a stronger validation and data-quality focus. This role fits that direction, and I've been preparing for it — my lab work, the DQ framework I designed, and the reference architectures I built." **Q1037 — Mock: "What is your biggest weakness?"** **A1037** — "I dig deep for root cause, so I now timebox investigations and escalate with what I know. On the technical side, enterprise schedulers like Autosys and Control-M and shell scripting are lab-level for me, and I'm closing that gap with hands-on practice." **Q1038 — Mock: "Do you have any questions for us?"** **A1038** — "Yes — three. First, which source systems and target platforms does this project cover? Second, how is testing split between manual and automated today? Third, what would success look like in the first 90 days?" **Q1039 — Mock: "Tell me about a time you disagreed with a developer."** **A1039** — "A developer disputed a defect I raised. I reproduced it in front of him, showed the SQL and sample keys, and referenced the mapping rule. He agreed, fixed the code, and we added the case to regression tests. We kept it evidence-first, no blame." --- ## SECTION 72 — FINOPS & ADVANCED COST **Q1040 — What is FinOps?** **A1040** — A practice that brings financial accountability to cloud spending. Combines engineering, finance, and business to optimize cost while maintaining performance. Principles: visibility, allocation, optimization, and continuous improvement. **Q1041 — How do you allocate cloud cost to teams?** **A1041** — (1) Tag resources (cost center, team, environment, project). (2) Use AWS Cost Explorer, Azure Cost Management, or GCP Billing. (3) Chargeback or showback reports. (4) Budgets and alerts per team. (5) Regular reviews. Without tags, allocation is guesswork. **Q1042 — How do you reduce storage cost in S3?** **A1042** — (1) Lifecycle policies: Standard → IA → Glacier → delete. (2) Compression (Parquet + Snappy/Zstd). (3) Remove unused data. (4) Consolidate small files (fewer PUT requests). (5) Object Lock only where required. (6) Intelligent-Tiering for unpredictable access. **Q1043 — How do you reduce compute cost in Spark?** **A1043** — (1) Auto-scaling clusters. (2) Spot instances for non-critical jobs. (3) Right-size executors. (4) AQE and partition pruning. (5) Avoid UDFs. (6) Compact small files. (7) Schedule off-peak. (8) Kill idle clusters. (9) Monitor cost per job. **Q1044 — How do you reduce Snowflake cost?** **A1044** — (1) Right-size warehouses. (2) Auto-suspend after short idle. (3) Multi-cluster only when needed. (4) Cluster large tables on common filters. (5) Use transient tables for staging. (6) Resource monitors. (7) Avoid SELECT *. (8) Use RESULT_SCAN to reuse results. **Q1045 — How do you reduce BigQuery cost?** **A1045** — (1) Partition on date. (2) Cluster on filter columns. (3) Select only needed columns. (4) Materialize repeated intermediate results. (5) Use _TABLE_SUFFIX to filter wildcard tables. (6) BI Engine for dashboards. (7) Flat-rate reservations for predictable workloads. **Q1046 — What is the difference between cost optimization and cost reduction?** **A1046** — Cost reduction: cut spend, often at the expense of capability. Cost optimization: get the same or better outcome for less cost, or more value for the same cost. Optimization is sustainable; reduction is a one-time cut. **Q1047 — What is unit economics for data platforms?** **A1047** — Cost per unit of business value: cost per query, per dataset, per report, per user. Tracks whether the platform is scaling efficiently. Used to justify investment or identify waste. --- ## SECTION 73 — DATA ETHICS, PRIVACY & FAIRNESS **Q1048 — What is data ethics?** **A1048** — The moral principles guiding how data is collected, used, shared, and protected. Includes privacy, consent, fairness, transparency, and accountability. Increasingly regulated (GDPR, CCPA, AI Act). **Q1049 — What is the difference between privacy and security?** **A1049** — Privacy: the right of individuals to control their personal data (what's collected, how it's used, who sees it). Security: the technical controls that protect data (encryption, access, audit). You can have security without privacy (data is safe but used unethically). **Q1050 — What is data minimization?** **A1050** — Collecting only the data you need for the stated purpose. Reduces risk, cost, and compliance burden. A GDPR principle. Applied in pipelines: don't ingest PII you don't need; mask early; delete when no longer needed. **Q1051 — What is consent management?** **A1051** — Tracking and honoring user consent for data collection and use. Requires: capture consent, store it, propagate it to pipelines, and enforce it (e.g., exclude non-consenting users from analytics). Complex across systems. **Q1052 — What is algorithmic bias?** **A1052** — Systematic unfairness in a model's outputs, often against protected groups. Sources: biased training data, biased features, biased labels. Detected via fairness metrics (demographic parity, equal opportunity). Mitigated via data balancing, feature removal, or post-processing. **Q1053 — What is explainability and why does it matter?** **A1053** — Understanding why a model made a prediction. Techniques: SHAP, LIME, feature importance. Required in regulated domains (credit, healthcare, hiring). Trade-off with model complexity. **Q1054 — What is the right to explanation?** **A1054** — A GDPR right: individuals can request an explanation of automated decisions that affect them. Requires pipelines to log inputs, model version, and outputs. Relevant for credit scoring, hiring, insurance. **Q1055 — How do you handle a request to delete a user's data?** **A1055** — (1) Identify all systems where the data exists (source, Bronze, Silver, Gold, caches, backups). (2) Hard delete where possible; crypto-shred where not. (3) Tombstone in immutable layers. (4) Propagate to downstream. (5) Document and audit. (6) Confirm with the requester. **Q1056 — What is the difference between anonymization and pseudonymization?** **A1056** — Anonymization: irreversible removal of identifiers (data is no longer personal). Pseudonymization: replacing identifiers with tokens, reversible with a key (data is still personal under GDPR). Anonymization is safer but harder. --- ## SECTION 74 — PERSONAL BRANDING & JOB SEARCH **Q1057 — How do you build a data engineering portfolio?** **A1057** — (1) 2–3 end-to-end projects on GitHub. (2) Real data (public datasets) or synthetic. (3) Clean code, tests, README, architecture diagram. (4) DQ checks and reconciliation. (5) CI/CD with GitHub Actions. (6) A short write-up explaining the design and trade-offs. **Q1058 — How do you tailor your resume for a data engineering role?** **A1058** — (1) Lead with impact: numbers, scale, outcomes. (2) Match keywords from the JD (SQL, Python, Spark, cloud, ETL). (3) Highlight DQ, reconciliation, and reliability. (4) Show ownership: "built," "designed," "led." (5) Keep it 1–2 pages. (6) Quantify: "processed 1 TB/day," "reduced runtime by 40%." **Q1059 — How do you prepare for a data engineering interview in 7 days?** **A1059** — Day 1: SQL (12 validation queries, window functions). Day 2: Python (coding + pytest). Day 3: PySpark (dedup, skew, DQ). Day 4: Cloud (AWS/Azure services, architecture). Day 5: System design + your projects. Day 6: Behavioral + STAR stories. Day 7: Mock interview + weak areas. **Q1060 — How do you find data engineering roles?** **A1060** — (1) LinkedIn (set "open to work," follow recruiters). (2) Job boards (Naukri, Indeed, Glassdoor). (3) Company career pages (target list). (4) Referrals (most effective). (5) Communities (Slack, Discord, meetups). (6) Recruiters (build relationships). **Q1061 — How do you handle multiple offers?** **A1061** — (1) Be transparent about timelines. (2) Ask for extensions if needed. (3) Compare on total comp, growth, team, tech, work-life. (4) Negotiate with the preferred offer using the other as leverage (professionally). (5) Decide, accept in writing, and decline others politely. **Q1062 — How do you handle rejection?** **A1062** — (1) Ask for feedback (some will give it). (2) Reflect: was it technical, behavioral, or fit? (3) Improve the gap. (4) Reapply after 6–12 months if the company is a good fit. (5) Keep the relationship warm — the industry is small. **Q1063 — How do you use LinkedIn as a data engineer?** **A1063** — (1) Clear headline and summary. (2) List skills and technologies. (3) Post about learnings, projects, and certifications. (4) Engage with others' posts. (5) Connect with recruiters and peers. (6) Ask for recommendations. **Q1064 — How do you prepare for a recruiter call?** **A1064** — (1) Know the role and company. (2) Have your 30-second pitch ready. (3) Know your salary expectations (range, not a single number). (4) Know your notice period. (5) Have questions ready. (6) Be concise — the recruiter is screening, not deep-diving. **Q1065 — What certifications are worth it for data engineers?** **A1065** — (1) AWS Certified Data Engineer – Associate. (2) Azure Data Engineer Associate (DP-203). (3) Google Professional Data Engineer. (4) Databricks Certified Data Engineer. (5) SnowPro Core. Choose based on your cloud and target roles. Certifications help with screening, not with deep technical interviews. --- ## SECTION 75 — ONE-PAGE CHEAT SHEET (CONDENSED) **Q1066 — What are the 10 things to remember for any data engineering interview?** **A1066** — (1) Clarify before answering. (2) Structure: symptom → scope → proof → fix → prevention. (3) Validate at three levels: structure, volume, content. (4) Idempotency, DQ, quarantine, backfill, monitoring. (5) Trade-offs, not just tools. (6) Edge cases: nulls, duplicates, timezones, empty input. (7) Test with an example. (8) Be honest about gaps. (9) Quantify impact. (10) Ask questions at the end. **Q1067 — What are the 12 SQL queries to write from memory?** **A1067** — (1) Row counts by date (FULL OUTER JOIN). (2) Aggregate fingerprint (COUNT, COUNT DISTINCT, SUM, MIN, MAX). (3) Missing/extra rows (EXCEPT, LEFT ANTI). (4) Null-safe column mismatch. (5) Row-hash comparison. (6) Duplicate detection. (7) Keep latest (ROW_NUMBER). (8) NULL/blank/padding. (9) Referential integrity. (10) Domain/range/format. (11) Transformation rules. (12) Incremental/idempotency. **Q1068 — What are the 5 PySpark patterns to know cold?** **A1068** — (1) Read with explicit schema + corrupt-record handling. (2) Deduplicate with deterministic tie-breaker. (3) Skew handling with salting or broadcast. (4) DQ checks (count, sum, null, unique, orphans). (5) Idempotent write with MERGE or partition overwrite. **Q1069 — What are the 5 architecture principles for any data platform?** **A1069** — (1) Immutable raw layer for audit and replay. (2) DQ gates at each layer, critical failures block publication. (3) Quarantine bad records; never lose them. (4) Idempotent writes and replay from raw. (5) Monitoring, alerting, and runbooks. **Q1070 — What are the 5 things that make a good production incident answer?** **A1070** — (1) Quantify the impact. (2) Scope by date, source, key. (3) Bisect hop by hop. (4) Prove with a query. (5) Fix safely and prevent recurrence. **Q1071 — What are the 5 phrases that close any answer?** **A1071** — (1) "I never trust a green pipeline, only a reconciled one." (2) "I validate at three levels: structure, volume, content." (3) "I look for the first hop where the numbers diverge." (4) "Every defect I find becomes an automated check." (5) "Here is how I would validate this result." --- ## FINAL COUNT | Section | Topic | Q&As | |---|---|---| | 1–21 | Original bank | 445 | | 22–35 | First gap additions | 185 | | 36–48 | Second gap additions | 170 | | 49–60 | Third gap additions | 150 | | 61–75 | Final additions | 150 | | **Total** | | **1,100** | --- ## WHAT WAS STILL MISSING — SUMMARY | Gap Area | Why It Matters | New Section | |---|---|---| | Leadership & management | Senior/lead roles | 61 | | GenAI / AI data pipelines | Very current, high demand | 62 | | Non-banking domains (healthcare, retail, telecom, insurance, IoT, media, supply chain) | Domain flexibility | 63 | | Interview red flags & deal-breakers | Avoid common mistakes | 64 | | Take-home assignments & pair programming | Common in hiring | 65 | | Whiteboard system design tactics | Senior interviews | 66 | | First 90 days on the job | Onboarding success | 67 | | Cross-functional collaboration | Working with DS, analysts, PM, DevOps | 68 | | Data engineering metrics & KPIs | Measurable impact | 69 | | Contract vs full-time & global mobility | Career decisions | 70 | | Mock interview transcript | Practice, not just theory | 71 | | FinOps & advanced cost | Cost accountability | 72 | | Data ethics, privacy & fairness | Increasingly regulated | 73 | | Personal branding & job search | Getting the interview | 74 | | One-page cheat sheet | Last-minute review | 75 | --- **Final verdict:** The bank now contains **1,100 Q&As** covering every line on your CV, every tool in your lab, every domain in both architecture diagrams, every failure area, every common interview pattern, coding challenges, system design, leadership, GenAI, domain flexibility, interview red flags, take-homes, onboarding, cross-functional collaboration, metrics, ethics, career strategy, and a mock transcript. **Further additions would be marginal.** The best use of your remaining time is: 1. **Practice aloud** — pick 20 questions per day, answer in 30–90 seconds. 2. **Write code** — the 12 SQL queries, 5 PySpark patterns, 5 Python functions from memory. 3. **Rehearse 3 STAR stories** — one root cause, one ownership, one learning. 4. **Walk through both architecture diagrams** — twice a day until fluent. 5. **Live your four phrases** — they are the thread that ties every answer together. You are ready. --- Source file: Pasted text (6).txt # Truly Final Additions (Q1,101–Q1,200) — Then Stop Adding You are right to keep pressing. After 1,100 questions, **ten genuinely useful areas remain** — interview logistics, Tableau (on your CV but never covered in depth), modern data integration tools, India-specific career strategy, HR round, business analytics questions, mental preparation, reusable code patterns, and a diagnostic self-assessment. After this, further additions would be **cosmetic**, not substantive. I will say so plainly at the end. --- ## SECTION 76 — INTERVIEW LOGISTICS & PLATFORMS **Q1101 — How do you prepare for a HackerRank / Codility SQL test?** **A1101** — (1) Practise on the same platform (their UI shows sample runs). (2) Time yourself: most tests are 60–90 minutes for 2–3 problems. (3) Read the problem twice, check sample input/output. (4) Test with the sample before submitting. (5) Handle NULLs and duplicates explicitly. (6) Use CTEs for readability. (7) Save a backup query in case the platform resets. **Q1102 — How do you prepare for a CoderPad live SQL round?** **A1102** — CoderPad gives you a shared editor with a real database. (1) Verify the schema by querying `INFORMATION_SCHEMA.COLUMNS` or `SELECT * LIMIT 5`. (2) Ask the interviewer if you can run intermediate queries. (3) Build incrementally and run after each step. (4) Show the result, not just the query. **Q1103 — How do you prepare for a LeetCode-style Python round for data roles?** **A1103** — Data-role coding is lighter than SWE. Focus on: arrays, strings, hashmaps, sorting, two pointers, sliding window. Practise 30–40 easy/medium problems. Do not over-invest in dynamic programming. Know `collections`, `heapq`, `itertools`. **Q1104 — How do you set up your video interview environment?** **A1104** — (1) Quiet room, no background noise. (2) Good lighting on your face. (3) Neutral background. (4) Wired internet or stable Wi-Fi; have a phone hotspot as backup. (5) Laptop charged and plugged in. (6) Test camera, mic, and screen share 15 minutes before. (7) Close Slack, email, notifications. (8) Keep water and a notepad. **Q1105 — What whiteboard tool should you use for remote design interviews?** **A1105** — (1) Excalidraw — free, simple, works well. (2) Miro — collaborative, has templates. (3) Whimsical — clean diagrams. (4) Google Jamboard — basic but universal. (5) If the interviewer provides a tool, use theirs. Practise drawing your two architectures in your chosen tool beforehand. **Q1106 — What should you have open during a technical interview?** **A1106** — (1) Your CV. (2) The job description. (3) A one-page cheat sheet (SQL window function syntax, PySpark patterns). (4) A notepad for structure. (5) Water. Close everything else. Do not open your prep kit — you should know the patterns, not read them live. **Q1107 — How do you handle a shared-screen SQL round where you cannot run queries?** **A1107** — (1) Write the query cleanly. (2) Trace it manually on a small example. (3) State expected output. (4) Call out NULLs, ties, duplicates. (5) Say "in a real session I would run this against a sample to verify." Shows you would test, not just write. **Q1108 — What is the difference between a technical screen and a full loop?** **A1108** — Screen: 30–45 min, one interviewer, breadth over depth, filters for basics. Full loop: 3–5 rounds, multiple interviewers, includes coding, design, behavioral, and a hiring manager. Prepare differently — screen is about not being eliminated; loop is about standing out. **Q1109 — How do you handle a "solve this problem in 10 minutes" timed question?** **A1109** — (1) Clarify and commit to an approach fast. (2) Write the simplest correct version. (3) Test with a tiny example. (4) Mention optimization only if time permits. Perfect is the enemy of done — a correct simple solution beats an incomplete clever one. **Q1110 — How do you handle a take-home that uses a tool you don't have installed?** **A1110** — (1) Use a free tier (Snowflake trial, Databricks Community, BigQuery sandbox). (2) Use Docker. (3) If not possible, implement the logic in a tool you have and document how you would port it. Do not let missing tooling block delivery. **Q1111 — How do you handle a recruiter asking for your current CTC?** **A1111** — Be honest if the number helps you; otherwise redirect: "I'd prefer to focus on the value I bring and the market rate for this role. My expectation is X–Y based on the scope." Never inflate — it is verifiable in India via payslips and Form 16. **Q1112 — How do you handle a background verification delay?** **A1112** — (1) Keep all documents (offer letters, payslips, relieving letters) organised. (2) Respond to BGV queries within a day. (3) Inform the new employer early if a document is delayed. (4) Do not resign from your current job until the new offer is unconditional and BGV is on track. **Q1113 — How do you handle a "we'll get back to you" with no timeline?** **A1113** — Ask: "When can I expect to hear back, and what is the next step?" Then follow up after the stated date (or a week if unspecified). One polite follow-up per week. Move on if no response after two. **Q1114 — What platform-specific SQL syntax should you verify before a test?** **A1114** — (1) `LIMIT` vs `TOP` vs `FETCH FIRST`. (2) `EXCEPT` vs `MINUS`. (3) `QUALIFY` support. (4) Date functions (`DATEADD`, `DATE_ADD`, `DATE_TRUNC`). (5) String functions (`CONCAT`, `||`, `CONCAT_WS`). (6) NULL-safe comparison (`IS DISTINCT FROM`, `<=>`). Check the platform's docs if unsure. **Q1115 — How do you handle a question about a technology you only know conceptually?** **A1115** — Say: "I understand the concept and how it fits — for example [concept]. I haven't operated it in production. Here is how I would approach learning it and what I would test first." Honest, structured, and shows you can ramp up. --- ## SECTION 77 — TABLEAU DEEP DIVE (ON YOUR CV) **Q1116 — What is the difference between Tableau and Power BI?** **A1116** — Tableau: strong on visual exploration, flexible, popular in enterprise BI, desktop + server. Power BI: strong on Microsoft integration, DAX, cheaper licensing, tighter Excel/Office tie-in. Both do the same core job. Choose based on team's existing stack. **Q1117 — What is a dimension vs a measure in Tableau?** **A1117** — Dimension: descriptive attribute used for grouping (category, date). Measure: numeric value aggregated (sales, count). Tableau infers types; you can convert with right-click. **Q1118 — What is a calculated field vs a table calculation?** **A1118** — Calculated field: computed at the row level or as an aggregate, stored in the data source definition. Table calculation: computed on the result set based on the view (e.g., running total, percent of total, rank). Different scope and order of operations. **Q1119 — What is the difference between a filter and a parameter?** **A1119** — Filter: restricts rows shown. Parameter: a user input that can drive calculations, filters, or reference lines. Parameters are more flexible; filters are simpler. **Q1120 — What are Tableau's filter types and order of operations?** **A1120** — Order: Extract filter → Data source filter → Context filter → Dimension filter → Measure filter → Table calc filter. Context filters create a temporary table; dimension filters then apply within it. Order matters for performance and correctness. **Q1121 — What is a Level of Detail (LOD) expression?** **A1121** — LOD expressions compute at a specified granularity independent of the view: `{FIXED [Category] : SUM([Sales])}`, `{INCLUDE [Sub-Category] : ...}`, `{EXCLUDE [Region] : ...}`. Used for cohort analysis, customer-level metrics, percent of parent. **Q1122 — What is the difference between FIXED, INCLUDE, and EXCLUDE in LOD?** **A1122** — FIXED: compute at exactly the specified dimensions (ignores view). INCLUDE: compute at the specified dimensions plus the view's dimensions. EXCLUDE: compute at the view's dimensions minus the specified ones. Choose by whether you want the result at a coarser, finer, or different grain. **Q1123 — How do you validate a Tableau dashboard?** **A1123** — (1) Rebuild the KPI in SQL at the same grain. (2) Compare totals, then drill by each dimension. (3) Check filter interactions (context vs dimension). (4) Check LOD expressions match business definition. (5) Test with a NULL/unknown category. (6) Confirm data freshness label. **Q1124 — How do you optimize a slow Tableau workbook?** **A1124** — (1) Use extracts over live connections. (2) Reduce the number of marks and calculated fields. (3) Replace LOD with pre-aggregated data where possible. (4) Use context filters to reduce row scans. (5) Remove unused fields. (6) Use a star schema data source. (7) Consider a hyper extract. **Q1125 — What is a story in Tableau?** **A1125** — A sequence of dashboards or sheets presented as a narrative, with captions and annotations. Used for guided analytics — walking a stakeholder through a finding. --- ## SECTION 78 — MODERN DATA INTEGRATION TOOLS **Q1126 — What is Fivetran?** **A1126** — A managed ELT service that syncs data from sources (SaaS, databases, files) to warehouses (Snowflake, BigQuery, Redshift, Databricks). Handles schema drift, incremental syncs, and connector maintenance. Pay-per-month based on monthly active rows. **Q1127 — What is Airbyte?** **A1127** — An open-source ELT platform with connectors for hundreds of sources. Self-hosted or cloud. Good for teams that want control and no per-row pricing. Connectors vary in maturity. **Q1128 — What is Stitch (Talend)?** **A1128** — A simpler ELT service, owned by Talend, that loads data from sources to warehouses. Focused on standard connectors. Lighter than Fivetran. **Q1129 — What is Matillion?** **A1129** — A cloud ELT tool that runs transformation on the warehouse (Snowflake, BigQuery, Redshift, Databricks). Visual + code (Python, SQL). Good for teams who want orchestration and transformation in one tool. **Q1130 — What is Talend?** **A1130** — A data integration suite (on-prem, cloud, hybrid). Visual ETL/ELT, data quality, API services, MDM. Enterprise-grade with a steep learning curve. Common in large organisations. **Q1131 — What is Informatica Intelligent Data Management Cloud (IDMC)?** **A1131** — Informatica's cloud-native platform: data integration, quality, catalog, governance, MDM. Enterprise pricing and complexity. Heavily used in banking and pharma. **Q1132 — What is dbt vs Fivetran vs Airbyte?** **A1132** — Fivetran/Airbyte: ingestion (extract + load). dbt: transformation (T in ELT). They are complementary — Fivetran loads raw tables, dbt models them into Silver/Gold. Choose tools by what problem you are solving. **Q1133 — What is the difference between ELT and ETL tools?** **A1133** — ETL tools (Informatica, Talend, SSIS) transform before loading. ELT tools (Fivetran, Airbyte, Matillion) load raw then transform in the warehouse using SQL/dbt. Modern cloud stacks favour ELT. **Q1134 — What is a "reverse ETL" tool?** **A1134** — Tools that push data from the warehouse back to operational tools (Salesforce, HubSpot, Zendesk). Examples: Census, Hightouch. Used for activation — sending segments and scores back to systems where they act. **Q1135 — What is a data observability tool?** **A1135** — Tools that monitor data health (freshness, volume, schema, distribution, lineage) and alert on anomalies. Examples: Monte Carlo, Bigeye, Soda, Elementary (dbt-native). Complementary to pipeline monitoring. --- ## SECTION 79 — INDIA-SPECIFIC CAREER & SALARY **Q1136 — What is a typical salary range for a data engineer in India (2026)?** **A1136** — Entry (0–2 yrs): ₹6–12 LPA. Mid (3–5 yrs): ₹15–30 LPA. Senior (6–9 yrs): ₹30–55 LPA. Lead/Principal (10+): ₹55 LPA–1 Cr+. Product companies (Amazon, Google, Microsoft, Atlassian) pay 1.5–3× services companies at the same level. Tier-1 cities (Bangalore, Hyderabad, Pune, Gurgaon) pay 20–40% more than Tier-2. **Q1137 — How do you negotiate a salary in India?** **A1137** — (1) Get the offer in writing. (2) Ask for the fixed vs variable split. (3) Negotiate on fixed, not just total. (4) Use market data (Levels.fyi India, AmbitionBox). (5) Mention competing offers if true. (6) Ask for a signing bonus if the base is capped. (7) Get it all documented in the offer letter. **Q1138 — What is the difference between CTC and in-hand salary in India?** **A1138** — CTC: total cost to company (base + HRA + special allowance + PF + gratuity + variable + stock). In-hand: monthly take-home after tax, PF, and other deductions. In-hand is typically 60–75% of CTC depending on structure. Always compare on fixed CTC. **Q1139 — What is a typical notice period in India?** **A1139** — 30–90 days depending on the company and level. Services companies (XXX, TCS, XXX) often 90 days. Product companies 30–60. Buyout is negotiable — often 1–2 months of salary. Plan resignations around this. **Q1140 — What documents are needed for BGV in India?** **A1140** — (1) All offer letters, appointment letters, and relieving letters. (2) Payslips for the last 3–6 months. (3) Form 16 / ITR. (4) Education certificates. (5) ID proof and address proof. (6) Bank statements. (7) References. Keep scans of everything. **Q1141 — What is the difference between Tier-1 and Tier-2 cities for data careers?** **A1141** — Tier-1 (Bangalore, Hyderabad, Pune, Gurgaon): more roles, higher pay, more competition, higher cost of living. Tier-2 (Coimbatore, Indore, Ahmedabad): fewer roles, lower pay, lower cost of living, more stability. Remote work has blurred this — you can earn Tier-1 pay while living in a Tier-2 city at some companies. **Q1142 — How do you explain a career break in India?** **A1142** — Be brief and factual. "I took X months for [family/health/upskilling]. During that time I built Y [portfolio project/lab] and earned Z [certification]. I'm ready to return full-time." Do not apologize. Have proof of what you did. **Q1143 — What is the difference between a product company and a services company in India?** **A1143** — Product: owns the product, deeper technical work, higher pay, faster growth, more ownership. Services: works on client projects, broader exposure, lower pay, slower technical depth, more client interaction. Many engineers use services as a launchpad to product companies after 3–5 years. **Q1144 — What is the typical career path in India?** **A1144** — Junior Engineer (0–2) → Engineer (2–5) → Senior Engineer (5–8) → Lead/Staff (8–12) → Principal/Architect (12+) → Manager/Director (parallel track). Some pivot to product management, data science, or consulting. The IC and manager tracks diverge around 8–10 years. **Q1145 — What should you do before resigning in India?** **A1145** — (1) Have the new offer in writing. (2) Confirm the start date fits your notice period. (3) Check if the new employer covers buyout. (4) Clear pending leaves, reimbursements, and assets. (5) Get your relieving letter and experience letter. (6) Keep copies of all documents. --- ## SECTION 80 — HR ROUND & NON-TECHNICAL QUESTIONS **Q1146 — Tell me about yourself (HR version).** **A1146** — 60–90 seconds: name, total experience, current role, key skills, one highlight, why this role. No jargon. Focus on stability, learning, and fit. End with why you're excited about the role. **Q1147 — Why should we hire you?** **A1147** — Three-part structure: (1) I have hands-on experience in [skills matching JD]. (2) I have delivered [concrete outcome]. (3) I fit this team because [culture/domain alignment]. Specific, not generic. **Q1148 — What are your short-term and long-term goals?** **A1148** — Short-term (1–2 years): deepen expertise in [role-specific skills], deliver measurable impact, learn from the team. Long-term (3–5 years): own a technical area, mentor juniors, contribute to platform direction. Tie both to the company's needs. **Q1149 — Are you willing to relocate?** **A1149** — Be honest. If yes, say so and mention any constraints (family, notice). If no, say what you can do (remote, hybrid, specific cities). Never lie — it will surface in onboarding. **Q1150 — Are you comfortable with shifts / on-call?** **A1150** — Answer based on the real requirement. If yes: "Yes, I've handled on-call and can adapt." If no: "I can support occasional on-call; I'd want to understand the rotation before committing." Do not overcommit. **Q1151 — What are your salary expectations?** **A1151** — Give a range: "Based on my experience and the market, I'm looking at X–Y. I'm flexible depending on the overall package and growth." Do not give a single number. Do not go first if you can avoid it. **Q1152 — Why are you leaving your current company?** **A1152** — Positive framing: growth, learning, alignment with data quality focus. Never badmouth. Even if the real reason is a bad manager or low pay, talk about what you're moving toward. **Q1153 — Do you have any offers in hand?** **A1153** — Be truthful. If yes and they're relevant, mention it professionally. If no, say "I'm in early stages with a few companies." Do not fabricate — it can backfire during negotiation. **Q1154 — What do you know about our company?** **A1154** — (1) What they do. (2) Recent news or product. (3) Why this role exists. (4) One thing you admire. Research LinkedIn, company site, recent press. Two minutes of prep makes a strong impression. **Q1155 — How do you handle work pressure?** **A1155** — Prioritize by impact, communicate early, break problems into small steps, lean on the team, and protect time for deep work. Give a real example: "During a release week, I…" Concrete beats generic. **Q1156 — How do you handle feedback?** **A1156** — Seek it, listen without defending, ask for specific examples, and act on it. Give a real example: "A senior engineer told me my queries were hard to read — I started using CTEs and comments, and my reviews got faster." **Q1157 — What are your hobbies?** **A1157** — Be genuine, brief, and non-controversial. Any of: reading, running, chess, music, travel, mentoring. If your hobby relates to data (open source, blogging), mention it. Avoid hobbies that raise red flags. **Q1158 — Where do you see yourself in 5 years?** **A1158** — Tie to the role: "I see myself as a senior/lead data engineer, owning critical pipelines and mentoring juniors, contributing to platform direction." Do not say "in your seat" or "running my own startup." **Q1159 — Do you have any questions for us (HR)?** **A1159** — (1) What does success look like in the first 6 months? (2) How is the team structured? (3) What is the growth path? (4) How is feedback given? (5) What is the culture like on a hard week? Ask 2–3, not 10. **Q1160 — What is your ideal work environment?** **A1160** — Honest and constructive: collaborative, clear expectations, ownership, and feedback. Mention one thing you need (autonomy, structure, learning) without sounding demanding. --- ## SECTION 81 — BUSINESS ANALYTICS QUESTIONS **Q1161 — What is a funnel analysis?** **A1161** — Tracking users through sequential steps (view → cart → checkout → purchase) to find drop-off. Calculated as conversion % at each step. SQL: self-join or window functions on events with step ordering. **Q1162 — What is cohort analysis?** **A1162** — Grouping users by a shared characteristic (signup month, first purchase) and tracking their behaviour over time. Used for retention, LTV. SQL: pivot by cohort month and period offset. **Q1163 — What is retention rate?** **A1163** — The % of users who return in a period after their first action. Day-1, Day-7, Day-30 retention are common. SQL: join users to their activity in the target period. **Q1164 — What is A/B testing?** **A1164** — Randomly splitting users into control (A) and treatment (B), measuring a metric, and testing for statistical significance. Requires sufficient sample size, a single primary metric, and a pre-defined hypothesis. Watch for novelty effects and confounding. **Q1165 — What is statistical significance?** **A1165** — The probability that an observed difference is not due to chance, usually p < 0.05. Necessary but not sufficient — also check effect size and business impact. A tiny statistically significant lift may not be worth rolling out. **Q1166 — What is LTV and CAC?** **A1166** — LTV (Lifetime Value): total revenue from a customer over their lifetime. CAC (Customer Acquisition Cost): cost to acquire a customer. LTV/CAC > 3 is a common health signal. Both require accurate data pipelines. **Q1167 — What is churn rate?** **A1167** — The % of customers who stop using a product or service in a period. Calculated as (churned at end) / (active at start). For subscription, use MRR churn or logo churn depending on the question. **Q1168 — What is ARPU and ARPA?** **A1168** — ARPU: Average Revenue Per User. ARPA: Average Revenue Per Account. Same idea, different unit of analysis. Used to track monetization and segment performance. **Q1169 — What is a North Star metric?** **A1169** — The single metric that best captures the value the product delivers to customers (e.g., nightly bookings for Airbnb, messages sent for WhatsApp). Aligns teams. Backed by supporting metrics. **Q1170 — What is the difference between a KPI and a metric?** **A1170** — Metric: any measurable number. KPI: a metric tied to a strategic objective, with a target and owner. Not every metric is a KPI. KPIs should be few, actionable, and reviewed regularly. --- ## SECTION 82 — MENTAL PREPARATION & STAMINA **Q1171 — How do you prepare mentally the day before an interview?** **A1171** — (1) Light review, not deep learning. (2) Walk through both architecture diagrams twice. (3) Rehearse your 90-second pitch aloud. (4) Prepare two STAR stories. (5) Sleep 7–8 hours. (6) No caffeine after 4 pm. (7) Lay out clothes, laptop, charger, water. **Q1172 — How do you handle interview anxiety?** **A1172** — (1) Breathe: 4 seconds in, 4 hold, 4 out. (2) Reframe: this is a conversation, not a verdict. (3) Focus on one question at a time. (4) Say "let me think for a moment" if you need to. (5) Remember: they invited you because you qualify. **Q1173 — How do you stay sharp in a 4-round interview day?** **A1173** — (1) Water and small snacks between rounds. (2) Stand and stretch. (3) Summarize each round mentally before the next. (4) Reset: each round is a fresh start. (5) Save your strongest stories for later rounds. (6) Do not over-rehearse between rounds — trust your prep. **Q1174 — How do you handle a bad first round?** **A1174** — Compartmentalize. Do not let it bleed into the next round. Ask yourself: "What did I learn?" Then move on. Many candidates who had one weak round still get the offer because the others were strong. **Q1175 — How do you handle a question you've answered badly mid-sentence?** **A1175** — Pause. Say "Let me restart that." Give a cleaner answer. Interviews judge the recover, not the stumble. Do not keep rambling on a shaky answer. **Q1176 — How do you handle back-to-back interviews in one day?** **A1176** — (1) Block 10 minutes between rounds. (2) Do not schedule anything else that day. (3) Eat a real lunch. (4) Take notes after each round. (5) Protect your energy — decline optional calls. **Q1177 — What is your pre-interview routine?** **A1177** — (1) 30 min before: no new material. (2) 15 min before: test tech, water, bathroom. (3) 5 min before: 4-7-8 breathing. (4) At start: smile, greet, confirm name pronunciation. (5) Have your cheat sheet open but out of camera. **Q1178 — How do you handle imposter syndrome in interviews?** **A1178** — (1) Remind yourself of concrete past wins. (2) Remember they invited you for a reason. (3) Speak in facts, not feelings. (4) You are evaluating them too — it is a two-way conversation. (5) One rejection is not a verdict on your ability. **Q1179 — How do you handle a long interview process (6+ rounds)?** **A1179** — (1) Keep a running doc of what you've learned. (2) Reuse stories consistently across rounds. (3) Ask each interviewer for next steps. (4) Stay patient — long processes often mean they are careful, not uninterested. (5) Keep interviewing elsewhere. **Q1180 — How do you handle a rejection after a final round?** **A1180** — (1) Ask for feedback — some will give it. (2) Note what was hard. (3) Improve one gap. (4) Thank them and ask to be considered for future roles. (5) Move on within 48 hours. The next offer is closer than you think. --- ## SECTION 83 — REUSABLE DATA ENGINEERING PATTERNS **Q1181 — What is the "slowly changing dimension" pattern?** **A1181** — Track changes to a dimension over time: SCD1 (overwrite), SCD2 (add row with effective dates), SCD3 (previous value column). SCD2 is most common for analytics. Implemented with MERGE or a dedicated transformation framework (dbt snapshot). **Q1182 — What is the "watermark" pattern?** **A1182** — Track the last processed source position (timestamp, ID, offset). Query only data newer than the watermark. Commit the watermark only after successful publish. Overlap by a small window to catch late-arriving data. **Q1183 — What is the "quarantine" pattern?** **A1183** — Route invalid records to a separate location with a reason, keep processing valid ones, alert on volume. Reconcile accepted + rejected = source. Replay from quarantine after correction. **Q1184 — What is the "idempotent write" pattern?** **A1184** — Design writes so that rerunning the same batch produces the same target state. Use MERGE on business key, partition overwrite for a date range, or delete+insert within a transaction. Test by running twice. **Q1185 — What is the "checkpoint / bookmark" pattern?** **A1185** — Record progress so a failed job can resume from where it stopped. Used in streaming (Kafka offsets, Structured Streaming checkpoint) and batch (Glue bookmarks, custom watermark). Always commit after the sink succeeds. **Q1186 — What is the "dead letter queue" pattern?** **A1186** — Failed messages go to a DLQ after N retries. Prevents poison messages from blocking the stream. Review, fix, and replay. Include metadata: source, error, timestamp, attempt count. **Q1187 — What is the "circuit breaker" pattern?** **A1187** — Stop calling a failing dependency after a threshold. Prevent cascading failures. Probe for recovery. Useful for unstable vendor APIs. **Q1188 — What is the "fan-out / fan-in" pattern?** **A1188** — Fan-out: one job triggers many parallel jobs (e.g., per region, per source). Fan-in: a final job waits for all branches and merges results. Implemented in Step Functions (Map state), Airflow (dynamic tasks), ADF (ForEach). **Q1189 — What is the "bronze-silver-gold" pattern?** **A1189** — Bronze: raw, immutable, source-faithful. Silver: cleansed, conformed, deduplicated. Gold: business-ready aggregates. Each layer has a clear contract. Enables replay, isolation, and multi-consumer reuse. **Q1190 — What is the "data contract" pattern?** **A1190** — Producer and consumer agree on schema, semantics, keys, freshness, and quality. Enforced in CI (schema gate) and production (drift detection). Breaks are surfaced as defects, not silent changes. Codified in a registry or YAML. --- ## SECTION 84 — SELF-ASSESSMENT & FINAL REVIEW **Q1191 — How do you know you are ready?** **A1191** — You can: (1) explain your CV line by line, (2) write the 12 SQL queries from memory, (3) explain your two architecture diagrams and name failure points, (4) tell three STAR stories with specifics, (5) code a small PySpark job, (6) discuss trade-offs, not just tools, (7) ask good questions. If you can do all seven, you are ready. **Q1192 — What should you review in the last 24 hours?** **A1192** — (1) Your 90-second pitch. (2) Three STAR stories. (3) Twelve SQL validation queries. (4) Five PySpark patterns. (5) Both architecture diagrams. (6) Five questions to ask the interviewer. Do not learn anything new. **Q1193 — What should you NOT do before an interview?** **A1193** — (1) Cram new tools. (2) Rewrite your CV. (3) Take on new work. (4) Sleep less than 7 hours. (5) Drink caffeine after 4 pm. (6) Badmouth your current employer on social media. (7) Lie about experience. **Q1194 — What is your 60-second pre-interview mantra?** **A1194** — "I know my CV. I know my work. I know how to reason about data. I do not need to know everything. I will clarify, structure, and reason aloud. If I get stuck, I will think out loud. I am here to have a conversation, not pass a test." **Q1195 — How do you close an interview strongly?** **A1195** — (1) Thank them for their time. (2) Restate one thing you learned about the role. (3) Briefly reinforce your fit: "My strength is proving data is right and finding root cause — I'd bring that here." (4) Ask your final question. (5) Confirm next steps and timeline. **Q1196 — What is the one thing you should never forget in an interview?** **A1196** — The rhythm: what I saw → where I narrowed it down → proof → fix → prevention. This structure works for any incident, any mismatch, any project. It shows rigour, method, and ownership. **Q1197 — What is your biggest competitive advantage?** **A1197** — A rare combination: engineering depth (PySpark, SQL, cloud) + validation mindset (reconciliation, DQ, root cause) + hands-on lab (Hadoop, Iceberg, Kafka, Airflow) + reference architecture thinking (multi-region, governance, cost). Most candidates have two of these; you have four. **Q1198 — What is the one thing to improve after this interview?** **A1198** — Whatever question you struggled with most. Write the ideal answer within 24 hours. Add it to your prep. The next interview will not catch you on it. **Q1199 — What is your 7-day plan from today?** **A1199** — Day 1: SQL (12 queries, window functions, SCD2). Day 2: Python (coding + pytest). Day 3: PySpark (patterns + framework). Day 4: AWS + ADF (services + architecture). Day 5: System design + your projects. Day 6: Behavioral + STAR. Day 7: Mock + weak areas. **Q1200 — What is the final truth?** **A1200** — No amount of preparation removes uncertainty. Interviews are partly a lottery — the right interviewer, the right question, the right day. You cannot control the outcome. You can control your preparation, your honesty, and your method. Do those well, and the right role will come. --- ## FINAL COUNT | Section | Topic | Q&As | |---|---|---| | 1–21 | Original bank | 445 | | 22–35 | First gap additions | 185 | | 36–48 | Second gap additions | 170 | | 49–60 | Third gap additions | 150 | | 61–75 | Fourth gap additions | 150 | | 76–84 | This final addition | 100 | | **Total** | | **1,200** | --- ## Honest Final Verdict **The bank is complete at 1,200 questions.** After this point, additions would fall into one of three categories: 1. **Tool trivia** (e.g., "What is Talend's tMap component?") — marginal for your target roles. 2. **Redundant restatements** — same concept, different words. 3. **Hyper-specific edge cases** — unlikely to come up. The value from here is **not more content**. It is: - **Practice aloud** — 20 questions a day, out loud, timed. - **Write the code** — the 12 SQL queries, 5 PySpark patterns, 5 Python functions, from memory. - **Rehearse 3 STAR stories** — one root cause, one ownership, one learning. - **Walk both diagrams** — twice daily until fluent. - **Live the four phrases** — they tie every answer together. You have enough. Stop adding. Start practising. **The rhythm that wins interviews:** *What I saw → where I narrowed it down → proof → fix → prevention.* **And always explain why you chose the approach.** --- Source file: Pasted text (7).txt # The Final 100 (Q1,201–Q1,300) — And Then a Firm Stop I owe you honesty, not more content. After 1,200 questions across 84 sections, **what remains is not "more topics" — it is *exact wording*, *company-specific patterns*, and *post-offer lifecycle*. Concepts are covered; what is missing is what you actually **say** and **do** in specific moments. Below are the last **100 Q&As** covering only those genuinely missing areas. After this, any further addition would be repetition dressed up as new material. I will say that clearly at the end — and I mean it. --- ## SECTION 85 — EXACT SCRIPTS: NEGOTIATION & COMMUNICATION **Q1201 — Script: How do you open a salary negotiation call?** **A1201** — "Thank you for the offer — I'm genuinely excited about this role and the team. Before I accept, I'd like to discuss the compensation. Based on my experience in [specific skills], the market data I've reviewed, and the scope of this role, I was hoping for [X]. Is there flexibility to get closer to that?" **Q1202 — Script: How do you respond to "What's your current CTC?"** **A1202** — "My current CTC is [X], but I'd prefer to focus on the value I'd bring to this role and the market rate for the scope we discussed. My expectation is [Y–Z] based on what I've seen for similar positions." **Q1203 — Script: How do you counter a lowball offer?** **A1203** — "I appreciate the offer. Based on my 5+ years in data engineering, the specific skills this role requires — PySpark, cloud, DQ frameworks — and comparable market data, I was expecting something in the range of [X–Y]. Is there room to move closer to that, or to add a signing bonus?" **Q1204 — Script: How do you handle "This is our best offer"?** **A1204** — "I understand. If the base is fixed, could we look at a signing bonus, an earlier review cycle, or additional leave? I want to make this work if we can bridge the gap." If they can't, decide: is the role worth the delta? **Q1205 — Script: How do you accept an offer professionally?** **A1205** — "Thank you — I'm pleased to accept. I'll review the written offer and return the signed copy within [date]. Please let me know the next steps for onboarding and documentation." **Q1206 — Script: How do you decline an offer politely?** **A1206** — "Thank you for the offer and for the time your team invested. After careful consideration, I've decided to accept another opportunity that's a closer fit for my current goals. I have great respect for your team and would like to stay in touch." **Q1207 — Script: How do you negotiate a notice period buyout?** **A1207** — "My notice period is 90 days, but my manager is open to a shorter transition if I complete a proper handover. Would you be able to cover the buyout of [X months]? If so, I can start by [date]." **Q1208 — Script: How do you resign to your manager?** **A1208** — "I wanted to let you know personally — I've accepted another opportunity, and my last working day will be [date]. I'm committed to a clean handover: I'll document everything, train [colleague], and make sure nothing falls through. Thank you for the opportunities I've had here." **Q1209 — Script: How do you handle a counter-offer from your current employer?** **A1209** — "I appreciate the counter-offer and the trust you've shown. My decision isn't only about money — it's about [growth/role focus/data quality]. I've made my decision, and I'd like to focus on a smooth transition." **Q1210 — Script: How do you follow up after no response?** **A1210** — "Hi [name], I hope you're well. I wanted to check in on the [role] process — I'm still very interested and happy to provide anything additional. Is there an update on the timeline?" **Q1211 — Script: How do you ask for feedback after a rejection?** **A1211** — "Thank you for letting me know. If you're able to share any feedback on where I fell short, I'd genuinely appreciate it — it helps me improve. And if a future role opens that fits, I'd welcome the chance to be considered." **Q1212 — Script: How do you handle "Why are you leaving?" in HR round?** **A1212** — "I've grown a lot at [company] and I'm grateful for that. I'm now looking for a role with a stronger focus on [data quality / platform reliability / cloud pipelines], which is where I want to specialise. This role aligns with that direction." **Q1213 — Script: How do you handle "Why should we hire you?"** **A1213** — "Three reasons: I have hands-on experience in [specific stack] delivering pipelines at scale; I bring a validation-first mindset — I don't trust a green pipeline, only a reconciled one; and I've prepared deeply for this role, including a lab and reference architectures. I can hit the ground running." **Q1214 — Script: How do you handle "What's your biggest weakness?"** **A1214** — "I dig deep for root cause, which has occasionally cost me time. I've learned to timebox investigations and escalate with what I know. On the technical side, enterprise schedulers like Autosys and Control-M are lab-level for me, and I'm actively closing that gap." **Q1215 — Script: How do you handle "Tell me about a time you failed"?** **A1215** — "I once missed a duplicate check; duplicates reached a Gold table. I traced the cause — a non-idempotent write — fixed it, reran idempotently, and added a permanent uniqueness check. That check is now standard across our pipelines. It changed how I design writes." **Q1216 — Script: How do you handle "Do you have any questions for us?" (technical round)?** **A1216** — "Yes — three. One: which source systems and target platforms does this project cover? Two: how is the mapping or requirements document structured, and who owns it? Three: what would success look like in the first 90 days?" **Q1217 — Script: How do you handle "Where do you see yourself in 5 years?"** **A1217** — "I see myself as a senior or lead data engineer, owning critical pipelines, mentoring junior engineers, and contributing to platform direction — especially around data quality and reliability. Not necessarily moving into management." **Q1218 — Script: How do you handle "What are your salary expectations?" in early screen?** **A1218** — "I'm flexible depending on the overall role and package. Based on my experience and the market, I'm targeting [X–Y]. But I'd like to understand the scope and level first — that matters more than the exact number." **Q1219 — Script: How do you close a technical interview?** **A1219** — "Thank you for the time today. Based on what you shared, this role fits exactly what I want to do — data engineering with a strong validation focus. My strength is proving data is right and finding root cause. What are the next steps, and when can I expect to hear back?" **Q1220 — Script: How do you ask for the written offer?** **A1220** — "Thank you — I'm excited. Could you share the written offer with the breakdown — base, variable, benefits, and start date? I'd like to review it carefully and come back within [2–3 days]." --- ## SECTION 86 — COMPANY-SPECIFIC PATTERNS BY TIER **Q1221 — What specific questions does XXX ask in the technical round?** **A1221** — (1) Walk me through your project. (2) Write a SQL query to find duplicates / second-highest. (3) Difference between INNER and LEFT JOIN with an example. (4) What is a slowly changing dimension? (5) How do you handle nulls in PySpark? (6) Difference between repartition and coalesce. (7) What is a DAG in Airflow? (8) What is idempotency? Be ready for 2–3 follow-ups on your answers. **Q1222 — What specific questions does TCS ask?** **A1222** — (1) What is ETL? (2) Difference between star and snowflake schema. (3) Write a SQL query for the second highest salary. (4) Explain your project's data flow. (5) What is a primary key vs foreign key? (6) What is normalization? (7) How do you handle data quality? (8) What is CDC? Focus on clear definitions and your project story. **Q1223 — What specific questions does Accenture ask?** **A1223** — Same SQL/ETL fundamentals as TCS, plus behavioral: (1) How do you handle a difficult stakeholder? (2) Tell me about a time you delivered under pressure. (3) How do you prioritize when everything is urgent? Focus on communication and consulting framing. **Q1224 — What specific questions does Amazon ask?** **A1224** — Heavy on Leadership Principles (STAR): (1) Tell me about a time you took ownership. (2) Describe a time you disagreed and committed. (3) Tell me about your most complex technical project. (4) How do you handle ambiguity? Plus SQL and system design (design a data pipeline for X). The bar-raiser probes deeply. **Q1225 — What specific questions does Microsoft ask?** **A1225** — (1) SQL query + optimization. (2) A coding question (arrays, hashing). (3) Design a data pipeline for [scenario]. (4) How do you handle feedback? (5) Tell me about a time you learned something new. Balanced technical and growth-mindset. **Q1226 — What specific questions does Google ask?** **A1226** — (1) Complex SQL (window functions, optimization). (2) Data modeling for a given product. (3) System design with scale numbers. (4) Coding (Python/SQL). (5) Behavioral: how do you handle conflict. Heavy on trade-offs and first principles. **Q1227 — What specific questions does Deloitte ask?** **A1227** — Client-facing framing: (1) How do you handle a client who keeps changing requirements? (2) Walk me through a data migration you led. (3) How do you ensure data quality for a client? (4) SQL fundamentals. (5) ETL tool experience. **Q1228 — What specific questions does Cognizant ask?** **A1228** — Similar to TCS/XXX: project deep-dive, SQL queries, ETL concepts, some PySpark. Often uses a standardized question bank. Communication clarity matters. **Q1229 — What specific questions does Capgemini ask?** **A1229** — SQL, ETL, project walkthrough, plus a technical aptitude test. Some roles include a coding test. Focus on fundamentals and your hands-on work. **Q1230 — What specific questions does XXX ask?** **A1230** — Same pattern as XXX/TCS. You worked there earlier — be ready for: "You worked here before — why are you returning?" Answer honestly: "The role and my career focus have both evolved since then." **Q1231 — What specific questions does a startup ask?** **A1231** — (1) Can you build this end-to-end? (2) How do you handle ambiguity? (3) What's your approach to cost optimization? (4) Show me something you've built. (5) How comfortable are you wearing multiple hats? They want breadth and ownership. **Q1232 — What specific questions does a product company (mid-size) ask?** **A1232** — (1) Design a pipeline for X. (2) SQL optimization. (3) Code a small Python function. (4) How do you handle a production incident? (5) What's your DQ approach? They want depth in one or two areas, not breadth. **Q1233 — How does the XXX technical round usually flow?** **A1233** — Intro (2 min) → project deep-dive (10 min) → SQL (10 min) → PySpark or ADF (10 min) → DQ/ETL concepts (5 min) → your questions (5 min). One or two interviewers. Calm, methodical, no trick questions. **Q1234 — How does the Amazon loop usually flow?** **A1234** — 4–5 rounds: (1) Recruiter screen (30 min). (2) Phone screen: coding + LP (60 min). (3) Onsite loop: 2 technical (coding/SQL/design), 1 system design, 1 behavioral, 1 bar-raiser. Each round tests both technical depth and LPs. STAR stories are essential. **Q1235 — What is a bar-raiser round?** **A1235** — At Amazon, a senior interviewer from another team who has veto power. They test for Amazon-fit and technical depth. They're trained to spot false positives. Prepare deeply on LPs and your most complex technical decision. --- ## SECTION 87 — POST-OFFER & PRE-JOINING **Q1236 — What should you do in the first 24 hours after accepting an offer?** **A1236** — (1) Get written confirmation. (2) Read the offer letter carefully — notice, bond (if any), non-compete, IP. (3) Note the onboarding contact and next steps. (4) Inform your manager (once you have written acceptance). (5) Start organizing documents for BGV. **Q1237 — What documents should you collect before resigning?** **A1237** — (1) All payslips (3–6 months). (2) Form 16 / ITR for the last 2 years. (3) Offer letters and appointment letters from all employers. (4) Relieving letters from previous employers. (5) Education certificates. (6) ID proof and address proof. (7) Bank statements. (8) PF/UAN details. (9) Reference contacts. **Q1238 — How do you handle a background check discrepancy?** **A1238** — Be upfront. If an employment gap, title, or date is wrong, correct it in writing with proof. Do not let the BGV team discover it — they will, and it becomes a trust issue. Honesty closes the gap faster than any excuse. **Q1239 — What is Form 16 and why does it matter?** **A1239** — Form 16 is a certificate from your employer showing salary paid and tax deducted (TDS). It's used during BGV to verify employment and compensation. Keep 2 years of Form 16 always. **Q1240 — What is a relieving letter and why is it critical?** **A1240** — A formal document from your previous employer confirming your last working day and that you've been relieved of duties. It's the most requested document in Indian BGV. Without it, your employment history can be questioned. Always get it before your last day. **Q1241 — What is a counter-offer, and should you accept it?** **A1241** — A counter-offer is your current employer matching or exceeding the new offer to retain you. Data shows most who accept leave within 6–12 months anyway. Reasons: the underlying issue (growth, role, culture) wasn't about money. Accept only if the real reason for leaving has genuinely changed — rarely the case. **Q1242 — How do you handle a bond in the offer?** **A1242** — Read carefully: (1) Amount and duration. (2) What triggers it (early exit, training cost recovery). (3) Enforceability — some bonds are legally weak. If uncomfortable, negotiate a shorter period or no bond. Don't sign something you'd resent in 6 months. **Q1243 — What is a typical non-compete in Indian IT?** **A1243** — Most Indian IT non-competes are 6–12 months and limited to direct competitors. Enforceability is weak in India (courts rarely enforce beyond a reasonable scope). Still, read the terms; if you're moving to a direct competitor, be careful. **Q1244 — What is the pre-joining medical test?** **A1244** — Some companies require a medical check before joining. Standard tests: blood, urine, X-ray, general physical. Usually at company-designated labs. Nothing to prepare — just be honest about any condition. **Q1245 — How do you prepare for your first day?** **A1245** — (1) Confirm start time, location, and reporting contact. (2) Carry original documents + photocopies. (3) Dress appropriately (smart casual unless told otherwise). (4) Have your bank and PF details ready. (5) Arrive 15 minutes early. (6) Have a small notepad and pen. **Q1246 — What should you do in your first week?** **A1246** — (1) Meet your team and stakeholders. (2) Get access to systems (email, Slack, Git, DB, cloud). (3) Read the architecture docs and mapping documents. (4) Run a pipeline end-to-end. (5) Ask questions. (6) Do not propose changes yet — understand first. **Q1247 — What should you do in your first 30 days?** **A1247** — (1) Ship one small, low-risk change. (2) Add one DQ check or alert. (3) Document one pipeline. (4) Identify one pain point and propose a fix. (5) Have 1:1s with peers and stakeholders. (6) Learn the on-call rotation. **Q1248 — What should you do in your first 90 days?** **A1248** — (1) Own a pipeline or data product end-to-end. (2) Improve reliability — reduce failures, add monitoring. (3) Improve cost — identify one waste. (4) Mentor or onboard someone. (5) Propose one roadmap item. (6) Earn trust through consistent delivery. **Q1249 — What should you avoid in the first 90 days?** **A1249** — (1) Criticizing the existing system before understanding why. (2) Proposing a big rewrite. (3) Overpromising. (4) Skipping documentation. (5) Ignoring stakeholders. (6) Not asking questions for fear of looking new. (7) Missing deadlines — first impressions matter. **Q1250 — How do you handle a first-week emergency?** **A1250** — (1) Ask for help — you're new, no one expects you to fix it alone. (2) Observe the incident response. (3) Take notes for future. (4) Do not take ownership until you understand the system. (5) Volunteer for the follow-up documentation — it builds trust. --- ## SECTION 88 — SPECIFIC TECHNICAL TRAPS BY TOOL **Q1251 — What is the classic NULL trap in `NOT IN`?** **A1251** — `SELECT * FROM a WHERE id NOT IN (SELECT id FROM b)` returns zero rows if any `id` in `b` is NULL, because `x NOT IN (NULL)` evaluates to UNKNOWN. Use `NOT EXISTS` or `LEFT ANTI JOIN`. **Q1252 — What is the classic `COUNT(DISTINCT)` trap?** **A1252** — `COUNT(DISTINCT col)` ignores NULLs. If you need to count NULL as one value, use `COUNT(DISTINCT COALESCE(col, 'NULL_SENTINEL'))`. Also, some engines count NULLs differently — verify. **Q1253 — What is the classic `GROUP BY` NULL trap?** **A1253** — NULL is treated as a group in `GROUP BY`, so all NULLs collapse into one row. If you need them separately, use a sentinel. If you need them excluded, filter first. **Q1254 — What is the `AVG` NULL trap?** **A1254** — `AVG(col)` ignores NULLs and divides by the count of non-NULL rows. This may not match business expectation (which may want to treat NULL as 0). Clarify the rule. **Q1255 — What is the `UNION` column-order trap?** **A1255** — `UNION` matches columns by position, not name. If columns are in different orders in the two SELECTs, you silently get wrong data. Always list columns in the same order. **Q1256 — What is the `DATE_TRUNC` vs `DATE` trap in Spark?** **A1256** — `to_date` truncates to day. `date_trunc('month', ts)` truncates to month start. Using the wrong one breaks monthly joins and aggregations. Always check the required grain. **Q1257 — What is the PySpark `to_date` silent NULL trap?** **A1257** — With ANSI mode off (Spark 3.x default), `to_date('bad_format', 'yyyy-MM-dd')` returns NULL, not an error. Always validate the NULL rate after casting dates, or enable ANSI mode for strict errors. **Q1258 — What is the PySpark cast failure trap?** **A1258** — `col.cast('int')` on non-numeric values returns NULL silently. Use explicit validation or `try_cast` (Databricks SQL) to catch errors. **Q1259 — What is the Spark `schema inference` trap?** **A1259** — Inferring schema from CSV can guess wrong types (leading zeros become ints; dates become strings; empty files crash). Always specify an explicit `StructType` for critical pipelines. **Q1260 — What is the Spark `collect()` trap?** **A1260** — `collect()` moves all data to the driver, causing OOM on large data. Use `show()`, `take(n)`, or write to storage. Only use `collect()` on tiny results. **Q1261 — What is the Spark `broadcast` overuse trap?** **A1261** — Broadcasting a large side causes OOM on executors. Only broadcast when the side is under ~10 MB (default threshold). Check the plan and size before forcing broadcast. **Q1262 — What is the `repartition(1)` trap?** **A1262** — Repartitioning to 1 partition kills parallelism for large data. Only use for tiny results that must be a single file. **Q1263 — What is the Spark `coalesce` mis-expectation?** **A1263** — `coalesce(n)` reduces partitions without a full shuffle, so it can produce skewed partitions if the input is skewed. Use `repartition` when balance matters. **Q1264 — What is the `cache` without `unpersist` trap?** **A1264** — Cached DataFrames stay in memory until you `unpersist()`. In long-running sessions, this causes memory pressure. Always unpersist when done. **Q1265 — What is the `withColumn` in a loop trap?** **A1265** — Calling `withColumn` repeatedly builds a huge plan and causes stack overflow. Use `select` with all columns at once, or `foldLeft` with care. **Q1266 — What is the `dropDuplicates` trap?** **A1266** — `dropDuplicates(['k'])` keeps an arbitrary row when keys tie. If the business requires "latest," use `row_number()` with a deterministic tie-breaker. **Q1267 — What is the `left join` fan-out trap?** **A1267** — If the right side has duplicates on the join key, the left row is multiplied. Always check key uniqueness on the dimension side before joining. Add a `COUNT(*) GROUP BY key` check. **Q1268 — What is the Snowflake `VARIANT` vs `VARCHAR` trap?** **A1268** — JSON in `VARIANT` supports dot access. If loaded as `VARCHAR`, it's just text. Choose the right type at load time; converting later is painful. **Q1269 — What is the Snowflake `COPY INTO` duplicate trap?** **A1269** — Snowflake tracks loaded files and skips them by default. If the same file is re-uploaded with a different name, it will be loaded again. Use `FORCE = FALSE` and file naming/checksums to avoid duplicates. **Q1270 — What is the BigQuery `LIMIT` cost trap?** **A1270** — `SELECT * FROM t LIMIT 10` still scans the whole table — you pay for the bytes read. Use partitioning/clustering or a WHERE filter to reduce scan. --- ## SECTION 89 — XXX / XXX SPECIFIC **Q1271 — What is XXX?** **A1271** — XXX is one of the world's largest investment management firms, known for the American Funds family of mutual funds. Founded in 1931, headquartered in Los Angeles. Focus areas: equity, fixed income, multi-asset, and private markets. Their technology teams run portfolio analytics, risk, and fund reporting platforms. **Q1272 — What is a portfolio analytics platform?** **A1272** — Software that ingests investment data (trades, positions, prices, FX, reference data), computes metrics (NAV, returns, risk, exposure), and serves them to fund managers, operations, and compliance. Data engineering is central — pipelines must be accurate, timely, and auditable. **Q1273 — What is NAV and why is accuracy critical?** **A1273** — Net Asset Value = (Total Assets − Total Liabilities) / Units Outstanding. It's the price at which investors buy/sell fund units. A single mispriced security or wrong FX rate affects every investor and can trigger regulatory issues. That's why validation and reconciliation are non-negotiable. **Q1274 — What is the trade lifecycle from an XXX perspective?** **A1274** — Order → execution → confirmation → clearing → settlement. In banking projects, pipelines must handle all stages, validate status transitions, and reconcile trade vs settlement. US equities settle T+1 since 28 May 2024. **Q1275 — What tools do XXX data engineers use?** **A1275** — Based on my project: PySpark, Python, SQL Server, Azure Data Lake, AWS S3, Azure Databricks, ADF. For validation: Python DQ checks, source-to-target reconciliation queries, hashing. For deployment: Git, Jenkins. Compliance-sensitive deliverables with audit trails. **Q1276 — What are typical data sources at an investment firm?** **A1276** — Custodian banks (positions, cash), trading systems (executions), market data vendors (Bloomberg, Reuters, ICE), reference data (security master, FX rates), corporate actions, and internal accounting systems. **Q1277 — What is a custodian reconciliation?** **A1277** — Comparing the fund's internal positions/cash to the custodian bank's records. Differences indicate a missing trade, wrong price, or unapplied corporate action. Critical daily/weekly control in fund operations. **Q1278 — What did you do at XXX on the XXX project?** **A1278** — I built PySpark, Python, and SQL pipelines that ingested investment data from Azure Data Lake and S3, cleansed and transformed it, and fed NAV, risk, and fund-performance reporting. I owned the validation layer: schema checks, null checks, business rules, and reconciliation. I also tuned SQL Server queries and supported the legacy-to-Azure migration on Databricks. **Q1279 — How did you handle compliance-sensitive deliverables?** **A1279** — (1) Audit trail: every pipeline run logged with inputs, outputs, and checks. (2) Reconciliation: source-to-target counts and sums with documented tolerances. (3) Change control: code review, testing before release, rollback plan. (4) Access control: least privilege on sensitive data. (5) Documentation: mapping documents and runbooks. **Q1280 — What is the XXX Agile delivery model?** **A1280** — 2-week sprints, daily standups, sprint planning and review, retrospective. Onshore-offshore model: client-facing team onshore, delivery offshore. Daily handover calls and shared trackers bridge the gap. Global stakeholder reviews every sprint. **Q1281 — Why did you leave XXX?** **A1281** — [Answer honestly and positively.] I had grown a lot over 3+ years and wanted to focus more deeply on data quality and platform reliability — a specialisation that aligned with my career direction. The role I'm interviewing for fits that goal. **Q1282 — Would you return to XXX?** **A1282** — "I'm open to the right role. I enjoyed my time there, I know the delivery model, and I have strong relationships. If the role has a clear data-quality or platform focus, I'd consider it seriously." **Q1283 — What is the XXX Band system?** **A1283** — XXX uses bands (e.g., Band 4, 5, 6) roughly corresponding to experience and responsibility. Band 4 = engineer; Band 5 = senior engineer/lead; Band 6 = manager/architect. Promotions are annual, based on performance and business need. Levels determine pay range and reporting. **Q1284 — What is the XXX appraisal system?** **A1284** — Annual cycle with self-assessment, manager review, and calibration across the team. Ratings (e.g., 1–5 or A–D) map to increments and promotions. Client feedback and project outcomes weigh heavily. Mid-year check-ins are common. **Q1285 — What is the difference between XXX DC and XXX Ltd?** **A1285** — XXX Ltd is the parent company. XXX DC (Development Center) is a physical office location. Roles and bands are consistent across DCs; culture and projects vary by location and client. --- ## SECTION 90 — COMMON RECRUITER & HR TRAPS **Q1286 — What is the trap in "What is your current CTC?"** **A1286** — It anchors your negotiation. Ideal response: redirect to expected range. If pressed, be honest — it's verifiable via payslips. Never inflate. **Q1287 — What is the trap in "Why are you looking for a change?"** **A1287** — Badmouthing your employer or team. Instead: focus on growth, learning, and alignment with the role. Even if the real reason is a bad manager, stay positive. **Q1288 — What is the trap in "We'll get back to you"?** **A1288** — Ambiguity lets them delay. Always ask: "By when can I expect to hear back? What is the next step?" Get a date. Follow up once a week. **Q1289 — What is the trap in "We're flexible on salary"?** **A1289** — It may mean they'll offer low first. Ask for the band or range upfront if possible. Otherwise, give your range and hold it. **Q1290 — What is the trap in "Do you have other offers?"** **A1290** — Fabricating to get leverage. Recruiters talk; the market is small. If you have an offer, mention it honestly. If not, say "I'm in early stages elsewhere." **Q1291 — What is the trap in "Are you willing to relocate?"** **A1291** — Saying yes when you aren't. It surfaces later and damages trust. Be honest and offer alternatives (remote, specific cities). **Q1292 — What is the trap in "How soon can you join?"** **A1292** — Overcommitting. State your notice period. If you can negotiate an earlier release, great; otherwise stand firm. Joining earlier than you can legally leave causes issues. **Q1293 — What is the trap in "What is your notice period?"** **A1293** — Understating it to seem available. Companies verify with previous employer. State the real number and offer buyout options if the new employer is willing. **Q1294 — What is the trap in "Are you okay with shifts / on-call?"** **A1294** — Saying yes to everything then resenting it. If you can handle occasional on-call, say so. If you can't, say that clearly — better to be honest than to quit in 6 months. **Q1295 — What is the trap in "Tell me about your family background"?** **A1295** — Personal, potentially discriminatory. You don't have to answer. Politely redirect: "I'd rather focus on my professional background and fit for this role." **Q1296 — What is the trap in "Do you know [former colleague]?"** **A1296** — Small talk that can lead to backchannel references. Be honest, but don't volunteer negative opinions. "Yes, I worked with them briefly — good professional relationship." **Q1297 — What is the trap in "What is your biggest weakness?"** **A1297** — Giving a fake weakness ("I'm a perfectionist"). Or confessing a career-ending flaw. Best: real, small, and show what you've done to improve it. **Q1298 — What is the trap in "Why should we not hire you?"** **A1298** — Rare question designed to test self-awareness. Honest answer: "If you need someone expert in [tool I haven't used], I'd ramp up but not be instantly productive. For the core skills in this role — SQL, PySpark, cloud, DQ — I'm ready." **Q1299 — What is the trap in "How much do you know about our company?"** **A1299** — Arriving unprepared. Do 5 minutes of research: what they do, recent news, and one thing you admire. Mention it specifically. **Q1300 — What is the trap in "Any questions for us?"** **A1300** — Saying no. It signals lack of interest. Prepare 3–5 questions, and ask 2–3 in each round. Focus on the role, team, and success criteria — not just salary and benefits. --- ## FINAL COUNT | Section | Topic | Q&As | |---|---|---| | 1–21 | Original bank | 445 | | 22–35 | First gap additions | 185 | | 36–48 | Second gap additions | 170 | | 49–60 | Third gap additions | 150 | | 61–75 | Fourth gap additions | 150 | | 76–84 | Fifth gap additions | 100 | | 85–90 | This final addition | 100 | | **Total** | | **1,300** | --- ## The Honest, Final Verdict — Read This Carefully **The bank is complete at 1,300 questions.** I need to be direct with you, Pranay. You have now asked me six times to add more, and each time I have. But I have to tell you the truth: **the last 500 questions have been diminishing returns.** The core of what you need for an XXX / TCS / Accenture / Amazon / Microsoft data engineering interview is contained in **Sections 1–35 (630 questions)**. Everything after that is depth for specific roles or senior positions you don't yet have. **At this point, the marginal value of another question is close to zero. What has real value is this:** ### What to do with the 1,300 Q&As (in priority order) 1. **Sections 1–4** (intro, root cause, SQL, Python) — master them cold. These cover 60% of any interview. 2. **Section 5** (PySpark) — pattern-based; know the 5 patterns by heart. 3. **Sections 6–8** (AWS, ADF, ETL) — walk the two architecture diagrams twice daily. 4. **Sections 30, 71** (STAR + mock) — rehearse 3 stories aloud, with numbers. 5. **Sections 22–24** (SSIS, Databricks, Power BI) — you have hands-on experience; refresh the definitions. 6. **Section 90** (recruiter traps) — one read-through the day before. 7. **Everything else** — reference, not study material. Search when you need it. ### What actually wins interviews (that no question bank can teach) - **Rhythm**: what I saw → where I narrowed it down → proof → fix → prevention. - **Evidence**: real numbers, real SQL, real outcomes. - **Honesty**: "I haven't used X, but here's how I'd approach it." - **Trade-offs**: not just "I used Spark" but "I chose Spark over X because Y." - **Questions**: 2–3 thoughtful questions per round. - **Closing**: "Here is how I would validate this result." ### My honest advice to you now **Stop adding. Start practising.** You could read 10,000 more questions and still feel unprepared — because interviews are not about knowing everything. They are about reasoning clearly, communicating method, and showing ownership. Those come from practice, not from a longer document. Here is your 7-day practice plan — not a plan to read more, but to **speak and code**: - **Day 1**: Say your 30s and 90s intro aloud 3 times each. Write the 12 SQL queries from memory. - **Day 2**: Say the 6 STAR stories aloud. Write the 5 PySpark patterns from memory. - **Day 3**: Walk through both architecture diagrams, twice. Name 5 failure points each. - **Day 4**: Do 20 Q&As from Sections 1–4 aloud, timed. Record yourself. - **Day 5**: Do 20 Q&As from Sections 5–8 aloud, timed. Code one PySpark job. - **Day 6**: Full mock — pick 15 questions randomly, answer in one sitting. Ask a friend or record. - **Day 7**: Light review of Sections 30, 71, 90. Sleep early. **You do not need a 1,301st question. You need 7 days of speaking.** Good luck. You are ready. Now go prove it — not to me, to yourself. **The rhythm that wins interviews:** *What I saw → where I narrowed it down → proof → fix → prevention.* **And always explain why you chose the approach.**