Are programmers in the USA still able to automate fetching their bank transactions, like was possible in the 1980s and 1990s?

Question

I have a pretty simple question for primarily programmers in the USA, but it also goes for those elsewhere.

I'm in Sweden. A couple of years ago, just when I had finally set up an automated system to fetch my bank transaction records automatically, the bank killed that entire "feature" (really a kind of abused "hole"), making it 100% impossible to automate keeping track of your finances, in turn forcing me to spend hundreds of active hours designing an elaborate and quite insane system to automate my personal bookkeeping as much as possible.

As I've just spent yet another day stomping out yet another bug in my system, and read old computer magazines from 1996 where this was all automated through dial-up modems and Windows/DOS desktop applications, I find myself once again wondering: is this some unique dumbed-down consumer slave mentality for Sweden, or is it the same now everywhere else? In particular, in the USA?

Are there millions of programmers out there who in recent years have been forced to design their own bookkeeping system, just like myself, and find ways to semi-automate the regular updating/verifying of this data by manually logging in with a dongle to their bank's website and manually downloading a CSV file, etc.?

OR do you have access to some sort of nice API after all? Every time I've asked them about this, I've either been ignored or they have implied that it's only available for large corporations and/or authorities, and nothing that I can ever get (or afford!) as as an individual. I assume that really rich people have access to this. I'm unsure if those "personal finance" software packages still work, and if so, what kind of API/interface they are using.

(They actually have an entire dedicated website called something like "Open API", but it's a lie. It turns out that it's not actually an "open API" whatsoever, and it's just for the big companies/authorities. This is not said upfront anywhere, and only became apparent after I had spent countless hours trying to decipher it.)

Answer 1

The US banks are notoriously bad when it comes to modernization of features. There are multiple ways of doing it and many different banks support many different ways, but there's no one standard all banks support in a similar way. Even the authentication is not standardized.

The "personal finance" packages (mostly) work, because they are developed by big corporations who can afford to develop for each bank specifically and pay their API access fees, which they get from consumers through subscription fees for their packages or through advertising to their "free" users, selling their financial data.

Answer 2

Fetch them through the Plaid API: https://plaid.com/products/transactions/

In summary, you give your bank account password to a third-party fintech app like Plaid, and it downloads the transactions for you and it makes them programmatically accessible to your scripts (via a REST API). It creates an API on top of your bank and allows you to write scripts that interact with the Plaid API to download your transactions.

Plaid does a "web scraping" technique under-the-hood where they download your bank statements, and then programmatically convert it into machine readable data. They make this data available over HTTPS for your scripts to download.

... Or web scraping:

You can do what Plaid does under the covers and write a Python script yourself that logs into your bank's website and navigates the HTML to download a statement and then convert that into data to import into your bookkeeping. Here's an example: https://www.neilgrogan.com/bank-tx-py/

There are security risks in these examples:

Giving out your password to fintech apps continues to be a difficult security risk to accept, but until the banks come up with more secure ways of downloading transactions through APIs, we're stuck with giving away passwords to fintech apps like Plaid if we want to automate things.

Answer 3

In the EU there is the PSD2 standard, which explicitly requires an API.

Many banks also (still) have the old German HBCI/FinTS standard that is used to access the accounts.

Answer 4

There is probably no standard.
However, at least two large banks I know offer APIs for money. You pay a monthly fee, and can use software to log in and download transactions all day long.

Answer 5

Banks provide digital bank account statements in form of files in formats like SWIFT MT940 or ISO 20022. These files are essential to automatize bookkeeping in companies which receive a lot of wire transfers every day. I've worked on some bookkeeping automation processes based on reading bank statement files in various formats myself (my work would not be of much use for you, though, unless you do your private bookkeeping with SAP ERP).

Unfortunately many banks will offer those to business customers only. But asking your bank if and how you can receive those files for your personal account doesn't hurt.

Answer 6

As a rule, API's for customer use are bad news for banks. The idea is noble enough but what is the worst that can happen?

Third parties that aggregate, ala Merrill Lynch, and insert themselves in the middle encourage credential sharing, and by extension, leaks.

You really cannot be supporting third party apps that you had no part in developing. API traffic may emanate from badly written and inefficient code, making loads hard to predict. Caching on the client side of historical data would be on an honour basis, for instance.

Sadly what this means for us is bandwidth hungry web applications which can be scraped to some degree, but end up being very volatile. Which is why the Merrill Lynch style aggregators failed. Too expensive to adapt to change in a timely manner.

Answer 7

There are a couple of different banking data aggregators that integrate the different open banking API's mandated by the EU PSD2 standard, for example Aiia and Tink (headquartered in Denmark and Sweden, respectively). Both of those allow you to create a free developer account to test out the service, and Tink specifically seems to have a €0.5 per user per month pricing tier with no extra costs that might fit well for a personal project like this.

Answer 8

I have been automating these things in the past, just like yourself. Many years ago, it was pretty easy to fetch all transaction data using open source libraries (i.e. GnuCash together with the separate library it uses for that purpose).

Over time, the two German banks I have business with changed or closed those possibilities. Right now, both in theory offer open TCP/IP ports where you can connect and at least the general protocols (HBCI/FinTS etc.) still seem to run; the protocol-level handshake still happens, so in some respect the lights are still on on their side.

But when I tried last time (in Q1/2021) for both banks I was not able to actually login and/or fetch any meaningful data anymore. I looked a lot, and you stumble across forums where people go on and on about how to use these features, with arcane settings only relevant to individual banks, and frequently changing without any notification from the banks of course. I never saw a bank which actually advertised these features (to private customers), I'm very sure you have zero chance to get any kind of useful information from their helpdesks. It seems pretty clear that it is an unwanted or at least unsupported feature, just swallowing human resources on the side of the banks, with no gain whatsoever for them.

Interestingly though, both banks offer manual download functionality in their online banking GUI where you can fetch your transactions at least month-wise in some format (not CSV/Excel) and import that in GnuCash, again manually. In theory one could probably automate this using GUI automation - i.e., something like AutoHotKey on Windows.

As an addendum, I used to automatically fetch stock prices from my broker way back by just scraping their website. Needless to say, in the meantime they are heavily gated behind modern features like single page web apps, XSS protection and so on (even for the basic "read-only" public information like this), so I kind of gave up on that as well; but I do assume this info would be more easy to get since it's pushed out on so many websites anyways. Just can't be bothered anymore.

Answer 9

In most parts of South East Asia API access to banking systems are by invitation only and you won't even get to talk to the people in charge of their computing systems unless you literally make thousands of transactions per day.

In all the projects I was involved in that had official permission to access bank data we were required to run our servers from inside a bank branch. If you're someone big and trusted like Quicken they may allow you to access their APIs from the internet.

Fortunately labor is cheap here. A lot of startups just manage user's money internally in a database and generates csv or xlsx files to upload to the bank. Someone is then hired to login to the banking portal and upload the generated files and download csv or xlsx from the bank.

For personal use there is practically no chance.

However most tech savvy people (including programmers) have learned to "program" the banking system using things like autodebit, standing instructions etc. to automate their personal finances. It's still programming but of a different kind - it's more like crafting clever Excel formulas.

Answer 10

Well, banks are in the business of making money; shocker!

I am 99.358% positive that banks have an active interest in not being on primetime news for:

MonkeyZeus found this weird loophole in the API, tune in tonight to see why banks hate him

Looking through the FAQs at https://www.chase.com/digital/data-sharing I can see why they don't want to give just anybody access to such power. As a developer I shudder at the thought of having to provide support to some script kiddie that can't figure out the difference between l and 1.

I think my bank allows me to download my transactions as a CSV file so if I was as adamant as you then I would use the CSV file.

Answer 11

The process can be automated using scraping software, e.g. Puppeteer or Selenium, however, as it has already been noted by other answers here, it requires above average maintenance efforts since scraping automation is susceptible to changes in the UI that you are scraping from. You may also reverse engineer back-end API network traffic in order to automate data retrieval, which could potentially be a more robust and compact solution, though it is similarly susceptible to breakage due to changes in API.

Of course, this is a pattern that extends to more than just banking data, but rather any data that a user produces is of real monetary value, and must be protected, even from the user itself. For example, Amazon does not make purchase history available to users older than a certain date. Law and regulation could potentially help, however, the more robust solution is moving towards DeFi, a new blockchain-driven model where financial services, e.g. insurance, investment, banking, etc. are made available in a peer-to-peer fashion, less susceptible to censorship and gatekeeping than traditional financial models.

Answer 12

Services vary by bank, not by country.

I would point out the world has changed slightly since the ‘90s. Before you get angry at the answer consider the bank’s position. Banks go to a lot of trouble to create high performing banking applications that obviate the need to even have branches in many cases. The needs of an enthusiastic technology hobbyist do not really rank in the overall scheme.

There are many APIs, but they are mostly used between FIs, aggregators and business accounts. Opening the standards to the public is an open invitation for criminals to hack away at an API at line speed.

Blame the cyber landscape, not the banks for lack of easy client facing APIs. If you like your bank but want better reports, use an aggregator or build a screen scraper.

Answer 13

Look into Selenium and other browser automation tools. In particular, have a look here:

https://github.com/angrykoala/awesome-browser-automation

Whatever your bank's website it like, you can probably scrape your data from it. And that's if they don't provide a CSV or even PDF for download, which would make the task as easy as "automate a browser to log in as you and fetch the data".

Answer 14

I'm fairly sure that I could automate the download of a CSV file from my bank, or alternatively web-scrape the most recent transactions shown after I log in.

What I couldn't do, by design, is log in automatically. Here in the UK, two-factor authentication is now mandatory. My password, plus a "something I have" token. It can be from a small electronic device supplied by the bank which looks like a credit card with buttons and a display, or an app on my mobile phone. I could probably automate a program which told me the bank's "challenge" and asked me to type in the response out of my authenticator gizmo.

I would be concerned if there was any bank anywhere holding my money, that did not use two-factor authentication. That's concerned enough to move my money somewhere safer!