Search - Code Extractor

function clean_text

Cleans and normalizes text content by removing HTML tags, normalizing whitespace, and stripping markdown formatting elements.

File: /tf/active/vicechatdev/improved_convert_disclosures_to_table.py

text-processing text-cleaning normalization html-removal markdown-removal

function clean_text_for_xml

Sanitizes text by removing or replacing XML-incompatible characters to ensure compatibility with Word document XML structure.

File: /tf/active/vicechatdev/improved_convert_disclosures_to_table.py

text-processing xml sanitization word-documents character-encoding

function extract_warranty_data_improved

Parses markdown-formatted warranty documentation to extract structured warranty data including IDs, titles, sections, disclosure text, and reference citations.

File: /tf/active/vicechatdev/improved_convert_disclosures_to_table.py

markdown-parsing text-extraction warranty-processing document-parsing regex

function clean_text_for_xml_v1

Sanitizes text strings to ensure XML 1.0 compatibility by removing or replacing invalid control characters and ensuring all characters meet XML specification requirements for Word document generation.

File: /tf/active/vicechatdev/enhanced_word_converter_fixed.py

text-processing xml sanitization data-cleaning word-documents

function quick_clean

Cleans flock data by identifying and removing flocks that have treatment records with timing inconsistencies (treatments administered outside the flock's start/end date range).

File: /tf/active/vicechatdev/quick_cleaner.py

data-cleaning data-quality flock-management livestock poultry

class ImprovedProjectVictoriaGenerator

Improved Project Victoria Disclosure Generator with proper reference management.

File: /tf/active/vicechatdev/improved_project_victoria_generator.py

class improvedprojectvictoriagenerator

function create_data_quality_dashboard

Creates an interactive command-line dashboard for analyzing data quality issues in treatment timing data, specifically focusing on treatments administered outside of flock lifecycle dates.

File: /tf/active/vicechatdev/data_quality_dashboard.py

data-quality dashboard interactive menu-driven timing-analysis

function show_critical_errors

Displays critical data quality errors in treatment records, focusing on date anomalies including 1900 dates, extreme future dates, and extreme past dates relative to flock lifecycles.

File: /tf/active/vicechatdev/data_quality_dashboard.py

data-quality validation error-reporting date-validation data-cleaning

function show_problematic_flocks

Analyzes and displays problematic flocks from a dataset by identifying those with systematic timing issues in their treatment records, categorizing them by severity and volume.

File: /tf/active/vicechatdev/data_quality_dashboard.py

data-quality reporting diagnostics livestock-management data-validation

function compare_datasets

Analyzes and compares two pandas DataFrames containing flock data (original vs cleaned), printing detailed statistics about removed records, type distributions, and impact assessment.

File: /tf/active/vicechatdev/data_quality_dashboard.py

data-quality comparison analysis reporting statistics

function parse_email_address

Parses email address strings by handling multiple addresses separated by semicolons and converting them to comma-separated format.

File: /tf/active/vicechatdev/msg_to_eml.py

email parsing string-manipulation formatting address-normalization

class FileCloudEmailProcessor

A class that processes email files (.msg format) stored in FileCloud by finding, downloading, converting them to EML and PDF formats, and organizing them into mail_archive folders.

File: /tf/active/vicechatdev/msg_to_eml.py

email-processing file-conversion cloud-storage filecloud msg-to-eml

function test_collection_creation

A diagnostic test function that verifies Chroma DB functionality by creating a test collection, adding a document, querying it, and cleaning up.

File: /tf/active/vicechatdev/test_chroma_collections.py

testing debugging chroma-db vector-database health-check

class ProjectVictoriaDisclosureGenerator

Main class for generating Project Victoria disclosures from warranty claims.

File: /tf/active/vicechatdev/project_victoria_disclosure_generator.py

class projectvictoriadisclosuregenerator

function test_incremental_indexing

Comprehensive test function that validates incremental indexing functionality of a document indexing system, including initial indexing, change detection, re-indexing, and force re-indexing scenarios.

File: /tf/active/vicechatdev/docchat/test_incremental_indexing.py

testing incremental-indexing document-indexing integration-test file-system

function main_v61

Command-line interface function that orchestrates the cleaning of ChromaDB collections by removing duplicates and similar documents, with options to skip collections and customize the cleaning process.

File: /tf/active/vicechatdev/chromadb-cleanup/main.py

cli command-line chromadb database-cleaning deduplication

function clean_collection

Cleans a ChromaDB collection by removing duplicate and similar documents using hash-based and similarity-based deduplication techniques, then saves the cleaned data to a new collection.

File: /tf/active/vicechatdev/chromadb-cleanup/main.py

data-cleaning deduplication chromadb vector-database similarity-detection

function load_data_from_chromadb

Connects to a ChromaDB instance and retrieves all documents from a specified collection, returning them as a list of dictionaries with document IDs, text content, embeddings, and metadata.

File: /tf/active/vicechatdev/chromadb-cleanup/main.py

chromadb vector-database data-loading document-retrieval embeddings

function main_v52

Command-line interface function that orchestrates a ChromaDB collection cleaning pipeline by removing duplicate and similar documents through hashing and similarity screening.

File: /tf/active/vicechatdev/chromadb-cleanup/main copy.py

cli command-line data-cleaning deduplication chromadb

function load_data_from_chromadb_v1

Retrieves all documents from a specified ChromaDB collection, including their IDs, text content, embeddings, and metadata.

File: /tf/active/vicechatdev/chromadb-cleanup/main copy.py

chromadb database document-retrieval vector-database embeddings

function setup_similarity_cleaner

A pytest fixture that creates and returns a configured SimilarityCleaner instance with a threshold of 0.8 for use in test cases.