המודל הסודי של IBM: איך Granite Vision מפענח כל מסמך עסקי?

So if we're talking about the LLM, we look at two benchmarks, the BFCL for tour calling and a IFF valve for instruction following. And we can see that granite vision is the 30 billion is rated top on top of Gemma and then the Granite Vision 8 billion. And the blue bars show all the Granite Vision versions. And also on the tool callings, we see very good results in terms of performance. So this is the baseline for what we use for Granite Vision. I just want to say that Granite Vision comes. This family of models comes with also other models for embedding for vision, speech, time series and so on. And for Granite Vision, we use the 3 billion LLM for as a baseline. So what do we mean when we say that we are focusing on enterprise workflows? So IBM as an enterprise, which was working mainly in the B2B area, we are very focused on business processes. Business processes include various artifacts, but mainly includes documents. When enterprises and businesses want their business processes to work well, they want usually to ingest documents that are sent by clients or by other enterprises. They want to be able to understand the documents, to read everything in them. And this is what we are focusing on. So when we talk about VLM for

business processes or workflows, we talk about aiming to have top performance, focusing on the few tasks on the left. So key enterprise task, we want to be able to understand charts that appear many times in business documents, like receipts, like procurement documents, like contractual documents, financial documents and so on. We want to be able to understand tables in the document and we want to be able to understand key value pairs. And we will show each of these tasks in a minute. And we want to do all that and beat up performance. Although we use a compact size, a very compact size for the model. So that requires technology innovation. It means that we need to have a very efficient architecture for the model. We also need to have training data and pipeline to generate synthetic data that we will be used to train our model to be efficient and effective. IBM has also an open source library for document extraction which is called DocLink. DocLink is a very, very popular open source, enjoying more than 500,000, more than half a million downloads a day. And we want Granite Vision to extend and expand DocLink so that DocLink can tackle tough cases and tough artifacts like tables, charts and key value pairs. And eventually, since we are coming from a research lab, we want all this work to have a lot of academic research. And indeed, what I show today has produced six papers in top quality conferences, as you can see, iclr, cvpr, ACL in a month and ICCV last year. So use cases. When I say tables, you can imagine a simple table as you usually see in papers. But the reality is that tables can be very, very challenging.

Tables can be with or without grid lines, table can have merge columns or merge rows, they can have multilayer headers, they can have empty cells and so on. Many times tables spill over one page and maybe more than one page. And all this creates challenges to understand and read the tables well and to be able to extract all the data accurately. In the bottom table that you can see, the colors were added by us. But you can see here a table which actually includes four tables. And this is really hard. And I can safely say that even frontier models don't read tables perfectly. It's very challenging. So we wanted to be very good at that.

We also want to be very good at charts. If we imagine documents, many times documents deliver information via charts. And charts can be very challenging as well. Right? So we have many, many types of charts, right? We have on the top left the radar chart and then three map charts and all the other types of charts. And these charts are coming in a diverse structure and in diverse colors. There are different ways to associate colors to labels, to associate data points to access. And, and this all makes it very challenging. And again, here, also here, frontier models don't do this perfectly and we Want it to be very good with charts as well.

And lastly, key value pairs. When I say key value pairs, I mean I look at contractual documents or business processes documents, and they all have data in them that, that we need to understand. Eventually. This is the form, and the form was filled in by someone. And we want to be able to understand all the keys and the values. For example, here in red, you can see that the key is a vehicle make and the value is Toyota Camry. And we have many of those. And it is really hard to understand all of this. This is relatively simple form, but many times key value pairs come in various structures. Sometimes the keys are above the value, but sometimes below the value, sometimes not even close to the value. And this brought us to understand that the only way we can do it here is to use a semantic view of the key value pairs to understand not just the structure, but also the content. And this allows us to relate keys and values and understand very complex documents by the vlm. So we needed to generate data for creating this model.

And the first thing that we did was tackling charts. So this work was presented in CVPR a couple of weeks ago. We created the chartnet pipeline. Chartnet pipeline uses a pipeline that creates from a seed chart. You can see on the top left, from each seed chart that we feed into the pipeline, we can create many, many charts. The way we do it is we use a VLM that creates for us the code that renders the seed charts. And then we ask a code LLM to augment the program that renders the chart in various ways, to use various types of charts, to use various colors, to use various positions for the labels, to use various values for the data points, and so on. And eventually we have augmented charts. This is done iteratively. Chartnet has 1.5 million charts in them. It also has a benchmark in it you can use to test yourself. And this is an example of one seed chart in orange that is coming into the pipeline from the left. And then all the generated charts, all the augmented charts that can be created from this one seed chart that we do, we also use. So this was used by Granite Vision. This was used to train Granite Vision actually to be very well, to do very well on charts. Just last week, mit, and I forgot to say that this work was done in collaboration with mit. So just last week MIT published and also in other places, you can see in the bottom some communication about this work, which is very interesting.

Varex is another work that we have done, another pipeline and benchmark that we created for this work. It was also presented in CPPR a couple of weeks ago. And Varex tackled the problem of complex documents that have many complex structures of key value pairs in them. And here we created strong 2,000, almost 2,000 documents, 20,000 fields in them, of business related documents with various structures of keys and values. As you can see on the left, this is an example. And this is how in this case, keys and values look like. This is a simple example, this is not a complex example. Because what we want to do is we want Granite Vision 4.1 to be able to understand key values from any type of form, not only the forms that it has trained on, not only the form that it has seen. And this means that we need to teach Granite Vision to understand the concept of key value pairs, the concept of related values to keys, the concept of hierarchical key value pair structures. And this is what we were doing using this Varx pipeline and benchmark to train Grindvision to do that. Okay, some words about the architecture. So what we do here is we take the input Image in Grindvision 4.1 and we use Lora fine tuned visual encoder.

We use SigLib2, which is trained together with the model. And then we go through keyformers. The qformers help us compress the visual tokens by a factor of four. And also each queueformer is looking at a separate tile. In that way we don't lose any important information, any important visual information in the special structure of the document. And then all the visual tokens are injected using Deep stack approach into the layers, into the relevant layers, depending on the level of the semantics of the token. For example, a token that is describing the whole document will be injected in the first layers, and so on. Granite Vision is based on Granite 4.1 micro, which is 3 billion. And and it is Lora fine tuned to create one model that we can use to serve only this one model, to serve and answer queries that are visually related, that have visual components in them, but also queries that are only language. So if we see a query which is only language, we can disable the adapters. If we see then a query which is using an input of an image or of a document, we can then enable the adapters again and serve only one model to answer this. Granite Vision 4.1 was trained on 100 million samples. It used 128H100 GPUs for two weeks. We estimated the cost as 70K for a run. And we did a lot of runs, of course, and on the left, on the bottom left you see a purple box which is called care.

CARE is another cairs, is another, another system or another model, a lightweight model that help us make the whole process more efficient. Here we must say that CARES is agnostic to the model itself. It can be used with Granite Vision, it can be used with other VLMs, it will be presented in ACL in a month. And what it does is it is predicting, looking at the query, the query prompt and looking at the image in the input. It is predicting the right resolution for the query and it's trying to choose the smallest resolution possible to be able to answer the query well. So you can of course imagine that if we have this dog here as an input, if we ask a query which is what is the breed of the dog, then we can probably get along with a small resolution, right? But if we ask a query which is what is the name of the dog which is written on the collar, then we will probably need a higher resolution than the other query. And this is done adaptively for each query, for each input, query and image. And you can see on the bottom right that the accuracy of the model using KERS is almost the same. We don't lose any accuracy almost at all, but we do save a lot of compute. So we save between 63% and 80% of compute using curs, which is quite nice, I think. So. Performance, if we look at performance, we want to be, as I said in the beginning, we want it to be very, very good, top of the line in tasks that are related to business documents. And this is a performance chart that shows the average on seven benchmarks, the two benchmarks that we created, ChartNet and VARX, but also other well known benchmarks like PubNet and so on and Pub

Tablenet and so on. And you can see here that Granite Vision 4.14B is rated first 86.5% accuracy on top of Cloud Opus 4.6. And below the dashed line you can see all the same class of size models like Gemma 4B and Quin 4B and Minstrel 8B and so on. So this is the average on 8, on 7, I'm sorry, on 7 benchmarks. And we were very happy to see this. It meant that we succeeded in our goal. And this is the types of things that GrantVision 4.1 can do. It can take a very complex box chart and extract it to a table which includes all the details, the right details in it. I can promise you that the details are correct. This is a benchmark that we did specifically for charts. So the previous one was an average on tables, charts and key value pairs. This is only for charts. And you can see that we have two tasks. The top task is a chart to CSV, which is an extraction task. And the bottom task is is chart to summary, which is a summarization task, of course. And you can see that on the top chart we have Granite Vision 4.14 B a bit below Gemini 3.1 Pro, but above GPT 5.4 and above Claudiopus 4.6. And on the bottom charts we can see that granovision is a bit below GPT 4.5.4 but above Claude Opus 4.6 and Gemini 3.1 Pro. And this is the type of charts that Granite Vision 4.1 does. Again, the extraction here is to a table. This is, you can see. I hope you can appreciate the challenge here and the complexity here of doing this. And all the numbers here are correct. It's in billions. This is why you see so many zeros. Okay. Because on the left axis you can see the numbers are on billions. And then a specific benchmark for tables. And also here we're using four very well known benchmark for tables. And you can see that Grande Vision 4.14B is rated very high on all of the benchmarks. On the top left you can see that it's a bit below Claude, but it's above GPT 5.4. And let me remind you, this is 4B. This model is 4B. It's not the size of GPT and Claude and same in the other benchmarks. Omnidoc bench is a bit more challenging benchmark. And Here Grand Vision 4.1 is below the Frontier models, but on top of all the other same size models from its class. And this is the last benchmark for kvp. This is Varx, what I mentioned before, a benchmark that we created and it's open source. And here we can see that Grialvision 4.1 is doing well. Not the best as we hoped, but is doing very well above Ministral and Gemma and Intern vl, but yes, below the Frontier models. And the reason for that is that this specific type of task, semantic KVP key value per extraction is very related to textual understanding and less to vision and less to the layout of the document. It is connected to the last of course, but it is very well related to the textual content. And this is why LLMs do this very well. And this is why all these frontier models are good here.

Just two last slides or three, maybe two exemplary use cases agentic workflow we want to be able to verify that the form that is ingested into the system includes all the details we use or many people use DocLink to extract the content. But now DocLink is expanded with Granite Vision and GranVision allows DocLink to understand the key value pairs and also on the right, when we prepare information for vector database in a rag task, we want to index all the chunks in the documents. We need Granite Vision to help DocLink to understand the special or important artifacts in the documents like charts and tables and key value so we are able to index them correctly in the vector database. So to summarize, Granite 4.1 Vision is an open source open weight model that everyone can use for business purposes. I think you remember everything that I said. It's only 20 minutes so I will and Chen is standing here and in a minute she's coming up on the board. So I will just say that we have two presentations here in IMVC today. Ellie is giving a presentation on adaptive resolution processing in vision language model and SRI is giving presentation on real world multimodal rag. And we also have three posters. One of them is about cares that I showed and another two and the team is here all day and we'll be happy to talk with everyone about what we did. Thank you.