🔍 Code Extractor

Loads all JSON documents from a designated documents directory and returns them as a dictionary indexed by document ID.

Purpose

This function scans a predefined DOCUMENTS_DIR directory for JSON files, loads each document using a helper function (load_document_from_file), and aggregates them into a dictionary. It's designed for bulk document loading at application startup or when refreshing the document cache. The function includes error handling and logging for operational visibility.

Source Code

def load_all_documents():
    """Load all documents from files"""
    documents = {}
    try:
        ensure_directories()
        for filename in os.listdir(DOCUMENTS_DIR):
            if filename.endswith('.json'):
                doc_id = filename[:-5]  # Remove .json extension
                document = load_document_from_file(doc_id)
                if document:
                    documents[doc_id] = document
        logger.info(f"📂 Loaded {len(documents)} documents from files")
    except Exception as e:
        logger.error(f"❌ Failed to load documents: {e}")
    return documents

Return Value

Returns a dictionary where keys are document IDs (derived from filenames without the .json extension) and values are the loaded document objects. Returns an empty dictionary if no documents are found or if an error occurs during loading.

Dependencies

• os
• json
• logging

Required Imports

import os
import json
import logging

Usage Example

# Prerequisites: Define required constants and helper functions
import os
import json
import logging

logger = logging.getLogger(__name__)
DOCUMENTS_DIR = './documents'

def ensure_directories():
    os.makedirs(DOCUMENTS_DIR, exist_ok=True)

def load_document_from_file(doc_id):
    filepath = os.path.join(DOCUMENTS_DIR, f'{doc_id}.json')
    try:
        with open(filepath, 'r') as f:
            return json.load(f)
    except Exception as e:
        logger.error(f'Failed to load {doc_id}: {e}')
        return None

# Usage
documents = load_all_documents()
print(f'Loaded {len(documents)} documents')
for doc_id, doc_data in documents.items():
    print(f'Document ID: {doc_id}')

Best Practices

• Ensure DOCUMENTS_DIR is properly configured before calling this function
• The function silently continues if individual documents fail to load, only logging errors
• Returns an empty dictionary on failure rather than raising exceptions, making it safe for initialization code
• Consider calling this function during application startup to pre-load all documents into memory
• The function depends on load_document_from_file() for actual document parsing, which should handle document validation
• Document IDs are derived from filenames by removing the .json extension, so ensure filenames follow this convention
• The function calls ensure_directories() to create the documents directory if it doesn't exist

Tags

Similar Components

AI-powered semantic similarity - components with related functionality:

• function load_document_from_file 72.9% similar

  Loads a document from a JSON file stored in a documents directory, deserializes it into a ComplexDocument object, and returns it.

  From: /tf/active/vicechatdev/vice_ai/complex_app.py

• function load_all_chat_sessions 65.8% similar

  Loads all chat session data from JSON files stored in a designated directory and returns them as a dictionary indexed by session ID.

  From: /tf/active/vicechatdev/vice_ai/complex_app.py

• function load_all_sessions 59.6% similar

  Loads all chat session data from JSON files stored in a sessions directory into memory on application startup.

  From: /tf/active/vicechatdev/docchat/app.py

• function load_database_schema 57.7% similar

  Loads a database schema from a JSON file and returns it as a DatabaseSchema object.

  From: /tf/active/vicechatdev/full_smartstat/sql_query_generator.py

• function save_document_to_file 57.1% similar

  Persists a document object to the filesystem as a JSON file, using the document's ID as the filename.

  From: /tf/active/vicechatdev/vice_ai/complex_app.py