How to extract tables from PDFs?

If you are tired of spending hours copying tables from PDFs, you’re not alone.

While PDFs are great for reading, they can be a nightmare when extracting tabular data. Whether it’s financial reports, invoices, healthcare bills, or research papers, extracting tables accurately and quickly can feel impossible.

But it doesn’t have to be. From simple tools like MS Excel to advanced solutions like AI-based IDP tools, there are multiple ways to extract tables efficiently and save hours of effort. Below is a quick summary to help you jump straight to your preferred method:

Several methods are available for extracting tables from PDFs, each with its own advantages and use cases. In this section, we will examine the six best methods for conversion in detail.

1. Using MS-Excel (Power Query)

Microsoft Excel is an accessible tool that you might already be familiar with. Excel’s recent versions offer a built-in feature Get Data (commonly known as Power Query) to import data from PDFs.

Here’s how you can use it to extract tables from a PDF:

Step 1: Navigate to the Data tab in an Excel Workbook.

Step 2: Select Get Data, then choose From File and then, From PDF.

Step 3: Excel will analyze the document and let you select which table to import.

This feature works well for simple tables but may struggle with complex layouts.

2. Using online PDF to table converters

Online converters are another quick and easy option you can use to extract tables from PDFs. These tools, such as Smallpdf, PDF2Go, and PDFTables, allow you to upload a PDF and download the extracted table in Excel or CSV format.

We will not cover all these tools in this section, but look at a general process one can follow for conversion. It is illustrated below with an example image.

Step 1: Visit the online converter’s website (for example: Smallpdf.com)

Step 2: Upload the PDF containing the table

Step 3: Select the output format (Excel or CSV)

Step 4: Download the converted file

3. Using a Python library (Camelot)

You can also use Python to extract PDF tables. We are taking an example of Camelot, which is a Python library.

We will break down the code into snippets and provide a detailed explanation for each part. At the end, you'll also find a unified code with a `main` function to download.

Step 1: Install necessary packages

Before starting, you need to install Camelot and Excalibur if they aren't already installed. 

pip install camelot-py[cv] excalibur-py

Step 2: Import necessary libraries

First, import the necessary libraries that will be used for PDF extraction as well as manipulating tabular data.

import camelot
import pandas as pd

Step 3: Extract tables from the PDF using Camelot

Camelot is a Python library that makes it easy to extract tables from PDF files. You can choose between two extraction flavours: `stream` and `lattice`. Here, we'll use `lattice` because it works well with tables that have clearly defined lines.

def extract_tables_from_pdf(pdf_path):

    # Read the PDF file and extract tables using Camelot
    tables = camelot.read_pdf(pdf_path, pages='all', flavor='lattice')
    
    # Checking if tables were extracted successfully
    if tables:
        print(f"Found {tables.n} tables in the PDF.")
    else:
        print("No tables found in the PDF.")
    
    return tables

Step 4: Display extracted tables

The display_tables function iterates over the extracted tables and prints each one to the console using pandasDataFrame's to_string method for better formatting.

def display_tables(tables):
    """
    Displays the extracted tables in the console.

    Args:
        tables (camelot.core.TableList): A list of tables extracted by Camelot.
    """
    for i, table in enumerate(tables):
        print(f"\nTable {i + 1}:")
        print(table.df.to_string(index=False))

Unified code:

Now, let's combine the code snippets into a single script with a main function that handles the entire process from reading a PDF to displaying the tables.

Run the script with your PDF file path specified in the `pdf_path` variable. This script will extract all tables from the PDF.

4. Using Tabula (Java-based tool with a Python wrapper)

Tabula is another popular tool for extracting tables from PDFs. Unlike Camelot, Tabula is a Java-based tool, but it can still be integrated with Python using the `tabula-py` library, which provides a Python wrapper around Tabula.

We will break down the process into detailed steps and provide a unified code with a `main` function that can be downloaded.

Step 1: Install necessary packages

Before starting, you need to install `tabula-py` and `pandas` if they aren't already installed. You should also have Java installed on your system since Tabula is a Java-based tool.

pip install tabula-py pandas

Step 2: Import necessary libraries

First, import the necessary libraries that will be used for PDF extraction and Excel writing.

import tabula
import pandas as pd

Step 3: Extract tables from the PDF using Tabula

Tabula is a tool designed specifically for extracting tables from PDFs. It can work in both GUI and command-line modes. When using it in Python, you can use `tabula.read_pdf()` to extract tables. 

Here’s how you can extract tables from a PDF using Tabula:

def extract_tables_from_pdf(pdf_path):

    """
    Extracts tables from a PDF file using Tabula.
    Args:
        pdf_path (str): The file path of the PDF.
    Returns:
        tables (list of pandas.DataFrame): A list of DataFrames extracted by Tabula.
    """
    
    # Read the PDF file and extract tables using Tabula
    tables = tabula.read_pdf(pdf_path, pages='all', multiple_tables=True)
    
    # Checking if tables were extracted successfully
    if tables:
        print(f"Found {len(tables)} tables in the PDF.")
    else:
        print("No tables found in the PDF.")
    
    return tables

Step 4: Display the extracted tables

The display_tables function iterates over the list of DataFrames and prints each one to the console using the to_stringmethod for a readable tabular format.

def display_tables(tables):
    """
    Displays the extracted tables in the console.

    Args:
        tables (list of pandas.DataFrame): A list of DataFrames extracted by Tabula.
    """
    for i, table in enumerate(tables):
        print(f"\nTable {i + 1}:")
        print(table.to_string(index=False))

If needed, pandas DataFrames can be converted to formats like, Excel, CSV or JSON for manipulation and processing.

Unified code:

Now, let's combine the code snippets into a single script with a main function that handles the entire process, from reading a PDF to extracting and displaying tables from it.

Run the script with your PDF file path specified in the `pdf_path` variable. This script will extract all tables from the PDF and display them.

5. Using Large Language Models (LLMs)

While Python-based tools such as Camelot and Tabula can be effective for extracting tables from PDFs, they require coding skills and are often limited in their scalability and flexibility. An alternative approach that leverages more advanced technology involves using Large Language Models (LLMs) to perform these tasks.

We will focus on using GPT-4 in two different methods:

1. Using the GPT UI
2. Using the GPT API

Method 1: Using the user interface

For this method, we will utilise GPT-4's user interface to convert a PDF document containing tables into an Excel spreadsheet. This method is ideal for users who prefer a more straightforward, no-code solution.

Step 1: Visit the GPT-4 user interface at chat.openai.com and log into your account.

Step 2: In the prompt section, upload the PDF file you want to convert and type a specific prompt to guide GPT-4 in extracting the tables.

Example prompts:

Method 2: Using the GPT API

For a more automated and scalable solution, you can use the GPT API to extract data from a PDF programmatically.

Step 1: Install the necessary libraries, including `openai` for interacting with the GPT API, `PyPDF2` for PDF extraction, and `pandas` for handling Excel operations.

pip install openai PyPDF2 pandas

Step 2: We’ll use PyPDF2 to extract the text from each page of the PDF. This extracted text will be processed to identify and extract tables.

import PyPDF2

def extract_text_from_pdf(pdf_path):
    """
    Extracts text from a PDF file using PyPDF2.
    
    Args:
        pdf_path (str): The path to the PDF file.
    
    Returns:
        str: The extracted text content.
    """
    with open(pdf_path, 'rb') as file:
        reader = PyPDF2.PdfReader(file)
        text = ""
        for page in reader.pages:
            text += page.extract_text()
    return text

Step 3: Create a detailed prompt instructing GPT-4 to identify and extract tables from the provided text. The tables will be returned as structured text (e.g., CSV-like format).

import openai

def extract_tables_from_text(text):
    """
    Extracts tables from the provided text using GPT-4.
    
    Args:
        text (str): The extracted text content.
    
    Returns:
        str: The extracted tables in text format.
    """
    prompt = (
        "Please extract all tables from the following text. "
        "Format them as CSV-like structures with columns and rows clearly defined:\n\n"
        f"{text}"
    )
    
    response = openai.Completion.create(
        model="gpt-4",
        prompt=prompt,
        max_tokens=2000,
        temperature=0.5
    )
    
    return response.choices[0].text.strip()

Step 4: Once the tables are extracted using GPT-4, they can be displayed, processed, or saved for further use.

def display_extracted_tables(extracted_tables):
    """
    Displays the extracted tables.
    
    Args:
        extracted_tables (str): The extracted tables in text format.
    """
    print("Extracted Tables:")
    print(extracted_tables)

Unified code:

Needless to say, you will need to replace "your_openai_api_key" with an actual value. Post that, you can run the script to parse tables from PDFs using GPT API.

By following these steps, you can effectively leverage LLMs to automate the process of extracting tables from PDFs and converting them into a CSV-like format which can be directly displayed or converted into other desirable formats. This process offers a modern alternative to traditional programming approaches.

6. Using AI-based IDP tools

Until now, we have looked at 5 different methods of extracting tabular data from PDFs to Excel. All of them present unique challenges.

The final approach that acts as the ultimate solution for building a scalable, robust process for converting tabular data from PDFs to a more accessible format is leveraging Intelligent Document Processing (IDP) tools.

They have OCR functionality built-in which means they are adept at recognising complex tabular structures, whether multiple tables of varied formats split across pages or nested tables with multiple line items.

They have advanced algorithms powered by generative-AI which can handle inconsistent cell formatting. What is more is that they offer post-processing options where you can perform formatting operations, such as, adding/removing characters, looking up against external tables or databases, in-app.

Not just that, they completely automate end-to-end workflows, including, importing PDF files, extracting tables, post-processing as discussed above, approval routing to verify accuracy and finally exporting to other software, all in one application.

We will be looking at one-such tool today, Nanonets, which provides a one-size-fits-all solution for handling the most complex tabular PDFs.

So, how do I get started?

Step 1: Visit app.nanonets.com. Sign up and log into your account.

Step 2: Click on “New Workflow” on the left panel and select “Pre-trained Tables Extractor.”

Step 3: Upload your PDF files and allow some time for processing.

Step 4: Once processed, you can click on the file to review the extracted data. Once satisfied, go back to the File Upload Screen, select the file and click on the “Download Extracted Data” button at the lower left corner of your screen. You can select the desired format in advanced settings on the dialog box that appears.

Now that we've discussed all the different methods to extract tables from a PDF, let's try to understand what is so different about PDFs that make it so challenging in the first place:

Why is table extraction from PDFs a huge challenge?

PDFs are immutable which means they cannot be modified or searched for specific data points. This renders the age-old trusty method of simply copying and pasting tables useless, even without taking into account its inefficiency.

But that’s barely scratching the surface when it comes to challenges of handling tabular data extraction from PDFs. So what are these challenges? Let’s take a look: 

1. Split tables across pages

Tables spanning multiple pages often have repeated headers or misaligned columns, making it difficult to reconstruct the complete table structure and maintain data integrity during extraction. 

Financial statements like balance sheets, cash flow statements, or even individual bank statements often have this problem of multiple tables that are occasionally split across pages, making it challenging to reconstruct complete financial data for investment analysis.

2. Inconsistent cell formatting

Varying font styles, sizes, and colours within a single table can be confusing leading to misinterpretation of data hierarchies and relationships between cells. 

Legal contracts, especially in real estate, often use varied formatting for clauses and sub-clauses within tables, hindering automated extraction of key terms and conditions for contract management systems.

3. Non-standard table structures

PDFs may contain nested tables, merged cells, or footnotes that can be challenging when it comes to correctly identifing table boundaries and cell contents. 

For instance, technical product catalogs across industries often use unconventional table layouts with hierarchical groupings, icons, and multi-line descriptions, challenging automated systems to correctly parse specifications, pricing, and inventory data for e-commerce platforms or procurement systems.

4. Background patterns and watermarks

Decorative elements or security features like watermarks can interfere with table detection and text recognition, resulting in incomplete or inaccurate data extraction. 

Bank statements and financial records often include security watermarks that interfere with OCR accuracy, complicating the extraction of transaction data for personal finance management tools or auditing processes.

Besides these common challenges, poor scan quality in scanned PDFs or data that isn’t explicitly delineated into rows and columns can also present additional roadblocks.

Final word

Choosing the right method or tool can make all the difference and save you hours of manual efforts. Whether you opt for simple solutions like MS Excel, advanced programming libraries like Camelot or Tabula, or scalable AI-powered tools like Nanonets, the key is finding what works best for your specific needs.

So, ready to tackle table extraction like a pro? Start with the method that suits your workflow, and watch how easily you can reclaim those lost hours.