审计上传的 CSV 中的缺失值与不一致问题,说明对业务的影响,并给出可用于生产的清洗管道。
我想要你充当一名资深数据科学架构师兼首席业务分析师。我正在上传一个包含原始数据的 CSV 文件。你的目标是执行一次深入的技术审计,并提供一个符合业务目标的、可用于生产的清洗管道。 请遵循以下 4 步执行流程: 技术审计与业务上下文:分析数据模式(schema)。识别不一致性、缺失值和数据异味(Data Smells)。简要说明这些数据问题可能如何影响业务决策(例如,日期不一致可能导致月度趋势分析错误)。 统计策略:基于审计结果,提出严格的插补策略(中位数 vs. 均值)、编码策略(独热编码 One-Hot vs. 标签编码 Label)和缩放策略(标准化 Standard vs. 鲁棒缩放 Robust)。 实现模块:编写一个模块化、符合 PEP8 标准的 Python 脚本,使用 pandas 和 scikit-learn。包含一个 Pipeline 对象,以便代码可直接用于 Streamlit 仪表板或自动化批处理任务。 后处理验证:提供断言检查以验证数据完整性(例如,检查空值,或通过降级类型转换实现内存优化)。 约束条件: 优先考虑内存效率(使用合适的 dtype,例如 int8 或 float32)。 如果存在目标变量,必须确保无数据泄露。 输出需采用结构化 Markdown 格式,并附有专业的代码注释。 我已上传文件。请开始审计。
I want you to act as a Senior Data Science Architect and Lead Business Analyst. I am uploading a CSV file that contains raw data. Your goal is to perform a deep technical audit and provide a production-ready cleaning pipeline that aligns with business objectives. Please follow this 4-step execution flow: Technical Audit & Business Context: Analyze the schema. Identify inconsistencies, missing values, and Data Smells. Briefly explain how these data issues might impact business decision-making (e.g., Inconsistent dates may lead to incorrect monthly trend analysis). Statistical Strategy: Propose a rigorous strategy for Imputation (Median vs. Mean), Encoding (One-Hot vs. Label), and Scaling (Standard vs. Robust) based on the audit. The Implementation Block: Write a modular, PEP8-compliant Python script using pandas and scikit-learn. Include a Pipeline object so the code is ready for a Streamlit dashboard or an automated batch job. Post-Processing Validation: Provide assertion checks to verify data integrity (e.g., checking for nulls or memory optimization via down casting). Constraints: Prioritize memory efficiency (use appropriate dtypes like int8 or float32). Ensure zero data leakage if a target variable is present. Provide the output in structured Markdown with professional code comments. I have uploaded the file. Please begin the audit.