39

I have a pretty simple question for primarily programmers in the USA, but it also goes for those elsewhere.

I'm in Sweden. A couple of years ago, just when I had finally set up an automated system to fetch my bank transaction records automatically, the bank killed that entire "feature" (really a kind of abused "hole"), making it 100% impossible to automate keeping track of your finances, in turn forcing me to spend hundreds of active hours designing an elaborate and quite insane system to automate my personal bookkeeping as much as possible.

As I've just spent yet another day stomping out yet another bug in my system, and read old computer magazines from 1996 where this was all automated through dial-up modems and Windows/DOS desktop applications, I find myself once again wondering: is this some unique dumbed-down consumer slave mentality for Sweden, or is it the same now everywhere else? In particular, in the USA?

Are there millions of programmers out there who in recent years have been forced to design their own bookkeeping system, just like myself, and find ways to semi-automate the regular updating/verifying of this data by manually logging in with a dongle to their bank's website and manually downloading a CSV file, etc.?

OR do you have access to some sort of nice API after all? Every time I've asked them about this, I've either been ignored or they have implied that it's only available for large corporations and/or authorities, and nothing that I can ever get (or afford!) as as an individual. I assume that really rich people have access to this. I'm unsure if those "personal finance" software packages still work, and if so, what kind of API/interface they are using.

(They actually have an entire dedicated website called something like "Open API", but it's a lie. It turns out that it's not actually an "open API" whatsoever, and it's just for the big companies/authorities. This is not said upfront anywhere, and only became apparent after I had spent countless hours trying to decipher it.)

20
  • 2
    Had to think of these questions: money.stackexchange.com/questions/131742/… Jan 26 at 21:27
  • 1
    I think some of us automate the file download using a headless browser. You could also check out plaid.com/en-gb/products/transactions. Jan 27 at 8:11
  • 7
    @BernhardDöbler I suspect that this question, that question, and the question linked in that question, are all asked by the same person. Jan 27 at 9:44
  • 13
    “forcing me to spend hundreds of active hours designing an elaborate and quite insane system” Nobody’s forcing you, and if it really cost hundreds of hours, I doubt it’s a win. xkcd.com/1205
    – Reid
    Jan 27 at 18:15
  • 2
    "[programmers] have been forced to design their own bookkeeping system" – who is forcing you to design your own bookkeeping system? Jan 28 at 22:46

14 Answers 14

28

The US banks are notoriously bad when it comes to modernization of features. There are multiple ways of doing it and many different banks support many different ways, but there's no one standard all banks support in a similar way. Even the authentication is not standardized.

The "personal finance" packages (mostly) work, because they are developed by big corporations who can afford to develop for each bank specifically and pay their API access fees, which they get from consumers through subscription fees for their packages or through advertising to their "free" users, selling their financial data.

1
  • 9
    Minor note: 'advertising to users', 'advertising to users using their private data' and 'advertising to users selling their private day' are 3 different things that are all done. Jan 27 at 12:53
23

Fetch them through the Plaid API: https://plaid.com/products/transactions/

In summary, you give your bank account password to a third-party fintech app like Plaid, and it downloads the transactions for you and it makes them programmatically accessible to your scripts (via a REST API). It creates an API on top of your bank and allows you to write scripts that interact with the Plaid API to download your transactions.

Plaid does a "web scraping" technique under-the-hood where they download your bank statements, and then programmatically convert it into machine readable data. They make this data available over HTTPS for your scripts to download.

making a transaction

... Or web scraping:

You can do what Plaid does under the covers and write a Python script yourself that logs into your bank's website and navigates the HTML to download a statement and then convert that into data to import into your bookkeeping. Here's an example: https://www.neilgrogan.com/bank-tx-py/

There are security risks in these examples:

Giving out your password to fintech apps continues to be a difficult security risk to accept, but until the banks come up with more secure ways of downloading transactions through APIs, we're stuck with giving away passwords to fintech apps like Plaid if we want to automate things.

5
  • 9
    I don't think password-only logins are legal in the EU, as it would be one factor authorisation?
    – gerrit
    Jan 27 at 16:57
  • 1
    As a pragmatic matter in the US, Plaid is the way to go. Plus, as a hobbyist, it will be free to you. Plaid also has some European support (with more "coming soon"), which might serve as a way to get around being too small to get direct PSD2 access.
    – josh3736
    Jan 27 at 20:47
  • 2
    @gerrit I am not sure about the law but all the banks here in Sweden requires some kind of two factor authentication, for example using a physical key or they allow using Mobile BankID which is a third party app that does not use the classic two factor authentication. Both Don't allow automation.
    – Rsf
    Jan 28 at 10:57
  • @gerrit Monzo is email magic link login, although because it’s an app that might count as 2 factors (device + email)?
    – Tim
    Jan 28 at 12:17
  • 2
    It's worth noting that 1) scraping may violate the terms of use for your bank's website (having a script login for you may as well--you'll need to check first!)--not all sites allow scraping, and 2) if you have a script login for you, you'll want to make really certain you have it handle your password securely, first and foremost meaning not storing your password in plaintext in the code or in a file read by the code. You really don't want your bank credentials to be compromised...
    – bob
    Jan 28 at 15:07
16

In the EU there is the PSD2 standard, which explicitly requires an API.

Many banks also (still) have the old German HBCI/FinTS standard that is used to access the accounts.

3
  • 7
    It seems that Sweden has PSD2 too. As far as I know, this system is not for hobbyists (i.e. you need a license before being admitted to it; probably a good thing, you don't want just ant Joe to make apps that control your banking). Having said that, some (especially newer, online-only) banks have APIs; E.g. take a look at this post by the Bunq bank.
    – Claude
    Jan 27 at 9:49
  • 1
    At least in germany the Banks I tried have a free tier or are free after some kind of registration. An access token is needed but only some kind of API calls may cost money. Furthermore you have a separtion of just looking at the account and transactions and starting a transaction, which always needs a 2nd authentification.
    – why.n0t
    Jan 27 at 11:43
  • PSD2 is only open for registered fintechs; every EU country has a registry of fintechs which have to be accepted by all the banks. You log in to the bank and give consent to the fintech's (AISP = Account Information Service Provider) access to your account data, and the fintech will get an OAuth2 Access Token that gives access for a number of days (30 or 90) until a new login is required. So you would have to find a registered PSD2 AISP that offers a REST API or that offers the bookkeeping you need directly. See e.g. creditmutuel.fr/oauth2/en/devportal/sca-workflows.html
    – Alex R
    Jan 28 at 21:29
7

There is probably no standard.
However, at least two large banks I know offer APIs for money. You pay a monthly fee, and can use software to log in and download transactions all day long.

6

Banks provide digital bank account statements in form of files in formats like SWIFT MT940 or ISO 20022. These files are essential to automatize bookkeeping in companies which receive a lot of wire transfers every day. I've worked on some bookkeeping automation processes based on reading bank statement files in various formats myself (my work would not be of much use for you, though, unless you do your private bookkeeping with SAP ERP).

Unfortunately many banks will offer those to business customers only. But asking your bank if and how you can receive those files for your personal account doesn't hurt.

5

As a rule, API's for customer use are bad news for banks. The idea is noble enough but what is the worst that can happen?

Third parties that aggregate, ala Merrill Lynch, and insert themselves in the middle encourage credential sharing, and by extension, leaks.

You really cannot be supporting third party apps that you had no part in developing. API traffic may emanate from badly written and inefficient code, making loads hard to predict. Caching on the client side of historical data would be on an honour basis, for instance.

Sadly what this means for us is bandwidth hungry web applications which can be scraped to some degree, but end up being very volatile. Which is why the Merrill Lynch style aggregators failed. Too expensive to adapt to change in a timely manner.

1
  • 2
    Almost every bank supports Mint. But probably Mint pays for the privilege. Jan 27 at 17:53
3

There are a couple of different banking data aggregators that integrate the different open banking API's mandated by the EU PSD2 standard, for example Aiia and Tink (headquartered in Denmark and Sweden, respectively). Both of those allow you to create a free developer account to test out the service, and Tink specifically seems to have a €0.5 per user per month pricing tier with no extra costs that might fit well for a personal project like this.

3

I have been automating these things in the past, just like yourself. Many years ago, it was pretty easy to fetch all transaction data using open source libraries (i.e. GnuCash together with the separate library it uses for that purpose).

Over time, the two German banks I have business with changed or closed those possibilities. Right now, both in theory offer open TCP/IP ports where you can connect and at least the general protocols (HBCI/FinTS etc.) still seem to run; the protocol-level handshake still happens, so in some respect the lights are still on on their side.

But when I tried last time (in Q1/2021) for both banks I was not able to actually login and/or fetch any meaningful data anymore. I looked a lot, and you stumble across forums where people go on and on about how to use these features, with arcane settings only relevant to individual banks, and frequently changing without any notification from the banks of course. I never saw a bank which actually advertised these features (to private customers), I'm very sure you have zero chance to get any kind of useful information from their helpdesks. It seems pretty clear that it is an unwanted or at least unsupported feature, just swallowing human resources on the side of the banks, with no gain whatsoever for them.

Interestingly though, both banks offer manual download functionality in their online banking GUI where you can fetch your transactions at least month-wise in some format (not CSV/Excel) and import that in GnuCash, again manually. In theory one could probably automate this using GUI automation - i.e., something like AutoHotKey on Windows.

As an addendum, I used to automatically fetch stock prices from my broker way back by just scraping their website. Needless to say, in the meantime they are heavily gated behind modern features like single page web apps, XSS protection and so on (even for the basic "read-only" public information like this), so I kind of gave up on that as well; but I do assume this info would be more easy to get since it's pushed out on so many websites anyways. Just can't be bothered anymore.

4
  • 1
    Weird. So how do German companies of all sizes get bank transactions into whatever accounting SW they use? Does every company need to have a person whose job description is to retype all transactions from online banking? Or do German online shops generally have a multi-week latency before the transfer you sent to pay for an order shows up on their monthly export and they can finally ship your package?
    – TooTea
    Jan 27 at 12:58
  • 1
    @TooTea By receiving their daily bank statements in form of standardized files in formats like ISO 20022, or in case of legacy-systems in Germany in particular with the format DTAUS which is only used in the German banking system (officially obsolete but still in use). I talked about this in my answer.
    – Philipp
    Jan 27 at 12:59
  • @TooTea, presumably they support it for companies; or the (non-free) software used by companies indeed does support the existing interfaces. I guess it makes a difference whether it's a big SAP system or Bob who wants to get his private bank statement.... I have improved the wording to make clear that I am talking about private customers.
    – AnoE
    Jan 27 at 15:25
  • "Does every company need to have a person whose job description is to retype all transactions from online banking? " Wouldn't surprise me in the slightest. Many many many things that are automated in UK or Denmark or the Netherlands are done manually with forms, in offices, by people here in Germany. Offices visits in person to do anything bureaucratic (heh heh). Germany is woefully UN-automated! An "online application" means also making an appointment - takes weeks , clerk prints it, types into another system, .... and then wait months, get a letter,make another appointment go, Q to pick up.
    – kpollock
    Jan 28 at 11:39
2

In most parts of South East Asia API access to banking systems are by invitation only and you won't even get to talk to the people in charge of their computing systems unless you literally make thousands of transactions per day.

In all the projects I was involved in that had official permission to access bank data we were required to run our servers from inside a bank branch. If you're someone big and trusted like Quicken they may allow you to access their APIs from the internet.

Fortunately labor is cheap here. A lot of startups just manage user's money internally in a database and generates csv or xlsx files to upload to the bank. Someone is then hired to login to the banking portal and upload the generated files and download csv or xlsx from the bank.

For personal use there is practically no chance.

However most tech savvy people (including programmers) have learned to "program" the banking system using things like autodebit, standing instructions etc. to automate their personal finances. It's still programming but of a different kind - it's more like crafting clever Excel formulas.

2

Well, banks are in the business of making money; shocker!

I am 99.358% positive that banks have an active interest in not being on primetime news for:

MonkeyZeus found this weird loophole in the API, tune in tonight to see why banks hate him

Looking through the FAQs at https://www.chase.com/digital/data-sharing I can see why they don't want to give just anybody access to such power. As a developer I shudder at the thought of having to provide support to some script kiddie that can't figure out the difference between l and 1.

I think my bank allows me to download my transactions as a CSV file so if I was as adamant as you then I would use the CSV file.

0

The process can be automated using scraping software, e.g. Puppeteer or Selenium, however, as it has already been noted by other answers here, it requires above average maintenance efforts since scraping automation is susceptible to changes in the UI that you are scraping from. You may also reverse engineer back-end API network traffic in order to automate data retrieval, which could potentially be a more robust and compact solution, though it is similarly susceptible to breakage due to changes in API.

Of course, this is a pattern that extends to more than just banking data, but rather any data that a user produces is of real monetary value, and must be protected, even from the user itself. For example, Amazon does not make purchase history available to users older than a certain date. Law and regulation could potentially help, however, the more robust solution is moving towards DeFi, a new blockchain-driven model where financial services, e.g. insurance, investment, banking, etc. are made available in a peer-to-peer fashion, less susceptible to censorship and gatekeeping than traditional financial models.

5
  • 1
    How would you automate logging in with two-factor authorisation?
    – gerrit
    Jan 27 at 16:59
  • @gerrit you just have to do two factor authentication once if you run the automation program from a single IP, so you could just leave this step manual since you’d have to perform 2FA once to get the program authenticated for its first time. Otherwise, you could experiment with e.g. buying a phone line you can automate and setting it as your number for 2FA.
    – Dean
    Jan 27 at 17:21
  • You need to do 2FA for every login, unless you plan on letting the scrapper run 24/7 to avoid logout. You cannot reverse engineer back end traffic easily since it involves encryption authentication and authorization at different levels, and anyway you will be stuck at the 2FA again.
    – Rsf
    Jan 28 at 11:06
  • @Rsf 2FA is not an issue. Read the above comment to learn why. And yes, reversing network traffic isn’t easy for beginners, but then again, programming done right isn’t easy.
    – Dean
    Jan 28 at 17:57
  • @Dean Funny, I’ve heard of test automation engineers in a Swedish bank that did exactly that, semi automatic automation to access the account. Technically it works but it’s not an elegant or fast solution
    – Rsf
    Jan 29 at 20:20
0

Services vary by bank, not by country.

I would point out the world has changed slightly since the ‘90s. Before you get angry at the answer consider the bank’s position. Banks go to a lot of trouble to create high performing banking applications that obviate the need to even have branches in many cases. The needs of an enthusiastic technology hobbyist do not really rank in the overall scheme.

There are many APIs, but they are mostly used between FIs, aggregators and business accounts. Opening the standards to the public is an open invitation for criminals to hack away at an API at line speed.

Blame the cyber landscape, not the banks for lack of easy client facing APIs. If you like your bank but want better reports, use an aggregator or build a screen scraper.

1
  • Services vary by bank and by country, Swedish banks offer Mobilt BankID logins in Sweden but not in other countries.
    – Rsf
    Jan 28 at 11:08
0

Look into Selenium and other browser automation tools. In particular, have a look here:

https://github.com/angrykoala/awesome-browser-automation

Whatever your bank's website it like, you can probably scrape your data from it. And that's if they don't provide a CSV or even PDF for download, which would make the task as easy as "automate a browser to log in as you and fetch the data".

2
  • 1
    Are you sure that this method is not against most banks' terms of service?
    – Flux
    Jan 28 at 8:44
  • You won't be able to pass the 2FA step with Selenium and the like
    – Rsf
    Jan 28 at 11:07
0

I'm fairly sure that I could automate the download of a CSV file from my bank, or alternatively web-scrape the most recent transactions shown after I log in.

What I couldn't do, by design, is log in automatically. Here in the UK, two-factor authentication is now mandatory. My password, plus a "something I have" token. It can be from a small electronic device supplied by the bank which looks like a credit card with buttons and a display, or an app on my mobile phone. I could probably automate a program which told me the bank's "challenge" and asked me to type in the response out of my authenticator gizmo.

I would be concerned if there was any bank anywhere holding my money, that did not use two-factor authentication. That's concerned enough to move my money somewhere safer!

4
  • 2FA is great for protecting the ability to make transactions, the convenience-vs-security tradeoff is quite different for read-only access.
    – Ben Voigt
    Jan 28 at 15:43
  • @Ben_Voigt Um. If J Random Phisher can easily get all details of my account, he is well armed for a social engineering attack. Also the balance alone makes it clear whether the attack is likely to be worthwhile. A large balance might indicate that a physical threat to the account holder might prove worth the risk of decades behind bars. Also he's just read my home address.
    – nigel222
    Jan 28 at 16:03
  • I never said read-only access should be allowed with no security at all. Anyway the reporting bureaus (e.g. ChexSystems) are selling all of that information to the attacker; if you are relying on its privacy for your security you have problems.
    – Ben Voigt
    Jan 28 at 16:10
  • @Ben Voigt the problem with some 2FA is that it is SMS based. Easily diverted. Tokens are OK, if not inconvenient, but huge numbers fail at once if you lose infrastructure from a third party supplier.
    – mckenzm
    Apr 21 at 5:39

Not the answer you're looking for? Browse other questions tagged or ask your own question.