83

In the era of big data, I find it surprising that banks and credit card companies only offer access to a ridiculously small number of transactions - often only your last 180 days, if that. The longest I've seen was 720 days going back. I suspect they do store everything, but intentionally limit access.

These transactions are text only, take up extremely little space, and storing an individual's lifetime worth of transaction would take less than 10MB of data; about the storage required by two MP3 files, and about 250k records per person per lifetime, if we generously assume everyone makes 10 transactions a day. But I'd be happy with only the last 10 years of transactions, so about 1MB per customer. One of the largest banks, JP Morgan Chase, has ~70MM credit card customers. That means 70TB of data for 10 years of records - hardly impressive for a corporation of that size, with $17B of net income in 2013, when 1TB of cloud storage costs $10 retail per month.

By comparison, would you put up with Gmail or any online email provider keeping only the most recent 120 days worth of email? (And emails do take a lot more space than transactions, are far more numerous, and have to be instantly retrievable.)

Storage requirements for transaction activity are trivial in an era where we're throwing around petabytes and zetabytes.

Is there a sound reason for banks not offering access to all your transactions, other than legacy software on their side?

Stupid limitation on the number of transactions

UPDATE: Somehow I hadn't found this 2011 Quora question asking the same thing.

Improve this question
12
  • 10
    I've wondered this myself. My credit union only shows me the last 365 days. It's frustrating at tax time, because I can't go all the way back to the previous year. It seems ridiculous to me that I need to keep all those paper statements.
    – Ben Miller - Remember Monica
    Oct 3, 2014 at 9:48
  • 4
    @DanDascalescu The amount of data you are estimating is way off the mark.
    – Dheer
    Oct 3, 2014 at 13:55
  • 5
    There's a few answers to this, as shown below. Best answers: 1. It will slow down the website. 2. They actually do take up a lot of storage space (not text only, like you say, there is a lot involved per transaction). and 3. They keep a certain amount of transactions (120 days or so) in a quick access database. Older transactions are kept elsewhere, and it is more costly to retrieve them. And 4. This usually isn't such a big deal that it is worth all the work to fix. Source: I work as a software developer in a credit union.
    – Bobo
    Oct 3, 2014 at 23:00
  • 4
    As a comment to @jl6: My (dutch) bank started offering historic transaction downloads several years ago and I can still download all transaction data from 2000 and later. Other banks here are recently also moving to increase the accessibility of the data. As such it is definitely possible from a technical point of view and just a matter of whether it has priority according to the bank.
    – Dennis
    Oct 6, 2014 at 8:52
  • 4
    Why is Enterprise Storage so expensive?
    – user
    Oct 8, 2014 at 12:21

11 Answers 11

Reset to default
65

"Things are the way they are because they got that way."
- Gerald Weinberg

Banks have been in business for a very long time. Yet, much of what we take for granted in terms of technology (capabilities, capacity, and cost) are relatively recent developments.

Banks are often stuck on older platforms (mainframe, for instance) where the cost of redundant online storage far exceeds the commodity price consumers take for granted. Similarly, software enhancements that require back-end changes can be more complicated.

Moreover, unless there's a buck (or billion) to be made, banks just tend to move slowly compared to the rest of the business world. Overcoming "but we've always done it that way" is an incredible hurdle in a large, established organization like a bank — and so things don't generally improve without great effort. I've had friends who've worked inside technology divisions at big banks tell me as much.

A smaller bank with less historical technical debt and organizational overhead might be more likely to fix a problem like this, but I doubt the biggest banks lose any sleep over it.

Improve this answer
5
  • 1
    That's exactly what we experienced before we switched our online shop backend provider from a large bank to Stripe.com (aka "smaller bank"). Banks hate customers. Stripe loves customers.
    – Uwe Keim
    Oct 8, 2014 at 12:21
  • 3
    @UweKeim "Banks hate customers" is absolutely not true. Otherwise they wouldn't be in business, or wouldn't banks (at least not the type we are referring to when we say "banks").
    – user253751
    Oct 9, 2014 at 9:01
  • 1
    @immibis I expect the remark wasn't meant to be taken so literally.
    – Chris W. Rea
    Oct 9, 2014 at 13:04
  • 3
    @ChrisW.Rea more accurate might be "Banks hate change, and customers love change." But presumably there are a number of banks that changed too quickly, and too riskily, and failed as a result.
    – user253751
    Oct 9, 2014 at 20:07
  • @immibis I don't think UweKeim was aiming to make an accurate comment, but simply expressing his personal frustration in dealing with large banks using hyperbole.
    – Chris W. Rea
    Oct 10, 2014 at 1:30
38

All the other answers here are correct, but I'll add one more perspective. I am a business architect at one of the world's largest retail banks. Every day I experience the frustration of trying to get large-scale corporate IT to do anything, so I feel that your question is just one facet of the wider question: "why are banks so old and busted?"

While it's true that the cost of online, redundant, performant, secure data storage is significantly higher than you anticipate in the question, it should still be well within the capacity of a large enterprise. The true cost is the cost of change.

Nothing at a bank is a green field development. Everything is a bolt-on to existing systems. Any change brings the risk that existing functionality will be affected, therefore vast schemes of regression testing (largely manually executed) spring up around even the most trivial developments. Costs scale exponentially with the number of platforms affected (often utterly distinct, decades-old, incompatible platforms that have arisen out of historical mergers and acquisitions). Only statutory, revenue-generating and critical maintenance change is approved.

Any form of cost-cutting that increases risk is quickly extinguished. This is because when things go wrong, IT get blamed by their business colleagues. This is because the business colleagues in turn get blamed by the regulators, the media, the customers, and the public at large. Who doesn't cuss their bank when the ATM is unavailable? The bank's IT organization develops a kind of management sclerosis, risk averse in the extreme. Banks can't ship a beta version and patch it later.

This ultra-low-innovation approach is a direct result of market and regulatory forces. If you were happy with a bank account that played fast and loose with your money the way Facebook plays with your data, then banking would be much cheaper, much more innovative, and much riskier.

Conclusion

To get back to your specific question, some banks actually do offer a much longer back catalog of transactions for download (usually only a few key fields of each transaction though), and the ones that don't most likely don't see it as a revenue generating selling point, and it therefore falls above their innovation appetite.

Improve this answer
18

I would say a lot of the answers here aren't quite right.

The main issue here is that banking is a highly oligopolous industry - there are few key players (the UK, for example, has only 5 major banks operating under a variety of brands: it's all the same companies underneath) and the market is very, very hard to enter owing to the immense regulatory burden.

Because the landscape is so narrow and it's possible to keep close tabs on all your competitors, there's no incentive to spend money on shiny new things to keep up with the competition - the industry is purely reactive. If nobody else has an awesome, feature-filled online portal, there's no need for any one bank to make one. If everybody is reactive, and nobody proactive, then it's a short logical deduction that improvements happen at a glacial pace.

Also take into account that when you've got this toxic "bare-minimum" form of competition, the question for these people soon turns to "what can we get away with?" which results in things like subpar online portals with as much information as you like delivered on paper for a hefty charge, and extortionate, price fixed administrative fees. Furthermore your transaction history is super valuable information. There are one or two highly profitable companies who collate international transaction data and whose sole job in life is to restrict access to that information to the highest bidders. Your transaction history is an asset in a multibillion dollar per year industry, and as such it is not surprising that banks don't want to give it out for free.

Improve this answer
7
  • 1
    Interesting point. Explains not only the answer to this question, but also why banks may have poor services in other aspects as well.
    – gerrit
    Oct 3, 2014 at 14:49
  • This is an interesting point, and applicable to Canada, too, where we have our "big 5" banks in a similar position of market dominance.
    – Chris W. Rea
    Oct 3, 2014 at 16:29
  • 1
    I also wonder how often most people require this infomation; I know I haven't ever needed it other than once to prove a transaction of buying an Xbox 360 within so many years. Those who value this infomation are still free to buy it from the bank for the surcharge they request for paper statements for archiving reasons.
    – djsmiley2kStaysInside
    Oct 3, 2014 at 22:45
  • 1
    @djsmiley2k I'm surprised at your comment. I think the answer, at least for the majority in the US, is ">= 1/year". This data is invaluable at tax time if itemizing deductions, and very few people get a larger return with standard deductions. This data will also be useful for tax audits, future budgeting, divorces, estate stuff, etc. To purchase this data every year for taxes from my bank would cost me almost $100, or some $5,000 over a lifetime.
    – Nicholas
    Oct 7, 2014 at 12:50
  • 1
    The bank would give me the information out for free, and only my information. I fail to see how keeping my information protected from myself brings the bank any revenue or benefit. The moment I see another bank that does give me the info, I'll jump, because all banks are lame anyway. I'm curious why Simple doesn't mention this benefit.
    – Dan Dascalescu
    May 8, 2015 at 4:40
15

One reason why they limit it is to protect you. If I hack your account, I get your entire financial history.

I can see a copy of every check you ever wrote. I can see the account number with every doctor, utility, and credit card. I can also see the account information on the back of those checks for all your relatives who you sent $10 for their birthday.

I can use the information in those accounts to see where you used to live, this allows me to spoof you when applying for new credit. If they ask if I ever lived on Main street in Anytown USA. I can confidently say yes.

If I only let you download a window of time, the responsibility is on you to protect that data that is before the window. They protect it in file isolated from the internet, and finally only in archive locations.

Some of the information doesn't exists in electronic form. Data from the 1990's and earlier may not exist in the form you want. They have been expanding the windows over time. I can see/download a pdf of my monthly statement going back 7 years. Of course that data can't go directly into quicken.

Some places do let you get a file that goes back farther, but they charge you for it, and it can only be done by them sending you the file. That prevents you from downloading your entire history everyday. That times 70 Million customers would overwhelm their server and other infrastructure.

Regarding the amount of data:

  • My quicken file with the last 10 years of data is about 60 MBytes and growing.
  • You also assume that the only data is in text files. They also store electronic copies of checks.
  • You also assume that the data you see on the transaction is the only data that exists regarding the transaction. They have fields that track the people who handled the transaction, the location of the transaction (ATM, teller, mail, scanned...), the information regarding when it was transferred to the other bank, when the funds were released...
Improve this answer
3
  • Interesting explanation, but as Dheer said, banks do retain data for longer periods for compliance purposes.
    – Dan Dascalescu
    Oct 3, 2014 at 10:17
  • 6
    But that data isn't available on line, which was the question. They keep it offline.
    – mhoran_psprep
    Oct 3, 2014 at 10:22
  • 2
    If a hacker had access to my bank account, leaking my financial records would be the least of my worries.
    – March Ho
    Feb 22, 2016 at 9:37
11

Although if you count only your data, it would be quite less 10 MB, multiply this by 1 million customers and you can see how quickly the data grows. Banks do retain data for longer period, as governed by country laws, typically in the range of 7 to 10 years.

The online data storage cost is quite high 5 to 10 times more than offline storage. There are other aspects, Disaster recover time, the more the data the more the time. Hence after a period of time Banks move the data into Archive that are cheaper to store but are not available to online query, plus the storage is not optimized for search. Hence retrieval of this data often takes few days if the regulator demands or court or any other genuine request for data retrieval.

Improve this answer
6
  • 12
    I find it hard to accept this as a realistic explanation in the age of Facebook, when they collect 500 TERABYTES per DAY.
    – Dan Dascalescu
    Oct 3, 2014 at 8:34
  • 13
    @DanDascalescu Facebook's business is storing and selling your data, not storing or lending money. They optimize for their business case, much as the bank for theirs.
    – Chris W. Rea
    Oct 3, 2014 at 13:17
  • 13
    10MB * 1 million is only 10 TB. That's less than $500 in storage costs, based on hard drive prices. That's on the totally unrealistic assumption that it doesn't need RAID or backups or anything of the sort.
    – ChrisInEdmonton
    Oct 3, 2014 at 14:55
  • 6
    You can pretty much guarantee that it's around $14500 per terabyte @Chris; SAN plus replication plus test environments get expensive very quickly... Should still be affordable though.
    – Ben
    Oct 3, 2014 at 16:14
  • 3
    Even if the banks maintain it but not online, why is is the offline data not available on request? I recently lost some bank data in a hard drive crash and asked my bank, and was told I would have to input the data from old statements (which are available as PDFs). If they have the old data in a machine-readable form, it should still be possible to get it to customers, even if it's not right there on the website.
    – BrenBarn
    Oct 3, 2014 at 18:51
9

To add technical detail to other answers, your (and some commenters') estimates of storing that data is woefully (many orders of magnitude) off.

Let's take your 10MB of transaction data per user.

  1. You're only estimating text records like in Quicken.

    Now add on the volume of storing everye check's image. That's 100K (if not 500Kb depending on resolution of the scan) per check. If you have 100 checks per year (not unrealistic, if you pay all utility/morgage bills by check, as well as purchases), you now have 10Mb/year to 50MB/year. Now you're asking for 10 years of this, so you have 100-500MB per customer. NOT 10MB-70MB as you initially assumed. Let's take a mid-range figure, 300 MB.

  2. You were estimating using consumer grate cheap-o storage (which Facebook can afford for their data, as they don't store transaction data). Now let's up that to enterprise server hard drives. Your storage costs just rose 2x-5x.

  3. Now, typically you'd have RAID. So 2x more.

  4. Most large financial institutions have multiple data centers. You typically store all data's copies in those data centers for DR purposes. Your multiplier added another 2x-4x

  5. Most production data servers have multiple copies (Write DB server + one or more read-only copies). Multiply by 2x-4x

  6. With some rare exceptions, most banks don't just have one central database server. Each major app / business line would have its own DB, so you multiply that by 2x-20x depending on the bank, especially if it's arrived at its size by merging with other banks and has dozens of inherited legacy systems.

  7. multiple backups. Regulatory backup requirements means you don't just back up your data once a year. You do it daily, till the data is purged from DB.

    Meaning, you don't store ONE copy of your transaction in backup. You stored, say, 10*365 copies, assuming 10 year retention)

So, at the low end, your cost estimates are 30*2*2*2*2*2 = 900 times off (3 orders of magnitude) just for live database storage, and 3500 times off for backup costs.

At the high end, they could be 50*5*2*4*4*20=16,000 times off (4-5 orders of magnitude)

At this range, no, it isn't worth it for the bank to keep your transactions available in DB and online any longer than bare-bones absolutely critically necessary.

Improve this answer
14
  • 3
    I don't know if it's true and the regulations are like that, but I find the thought of having to have thousands of backups of the exactly same data horrifying. Compressed, incremental backups should bring that number down severely. Also not sure if really all data centers need their own, complete backup.
    – mafu
    Oct 4, 2014 at 17:12
  • 4
    @cHao - This site is the wrong one for this discussion. If you want to know why that approach doesn't work for backing up databases (as opposed to filesystems), ask on DBA StackExchange.
    – user2932
    Oct 5, 2014 at 16:06
  • 3
    @DVK: You're stating that as if it were a known fact (which it isn't), to back up a couple of very questionable numbers. Well, those numbers are now being questioned. You don't get to pawn the justification off on DBA.se.
    – cHao
    Oct 5, 2014 at 16:40
  • 2
    @cHao - the fact that you don't back up individual rows in a table based on when those rows were inserted is DBA 101. It's just NOT how Sybase or Oracle backups work. You back up entire database, as a unit. It is a well known fact to anyone who ever touched a database backup.
    – user2932
    Oct 5, 2014 at 16:58
  • 3
    @cHao - one of them did go bankrupt. But that had more to do with subprime mortgages than backups. And I was part of the team resposible for company wide backups, so I know exactly what was and wasn't backed up. Transaction databases were backed up whole. Daily.
    – user2932
    Oct 5, 2014 at 17:40
5

Many good points have been brought up, and I'll just link to them here, for ease.

Source: I work at a credit/debit card transaction processing company on the Database and Processing Software teams.

1. Security

See mhoran_psprep's answer.

2. Tradition

See Chris' answer.

3. System Integrity

Believe it or not, banks don't expose their primary (or secondary) database to end users. They don't expose their fastest / most robust database to end users. By only storing x days of data in that customer-facing database and limiting the range of any one query, any query run against it is much less likely to cause system-wide slowness.

They most definitely have database archives which are kept offline, and most definitely have an employee-facing database which allows employees to query larger ranges of data.

4. No Added Value

What would a bank have to gain by allowing you to query a full year of transactions?

Improve this answer
4
  • My bank keeps exhorting me to sign up for electronic "statements". They are very proud that their website would allow me to access a whole 18 months worth of transactions. Yeah, a whole 18 months, that is something to be proud of. No thanks, keep sending me the paper. If they weren't so utterly ridiculous, they might convince me to sign up for electronic deliver. So, that is one thing they would have to gain.
    – jjanes
    Oct 4, 2014 at 4:56
  • "No added value?" Well, I'd be happier with my bank
    – bobobobo
    Sep 29, 2019 at 17:52
  • You don't need to expose any database if you simply have an API for requesting transactions. That API can enforce whatever limits it wants, and the database is kept safe from direct access.
    – Dan Dascalescu
    Dec 7, 2019 at 15:16
  • @jjanes Maybe its not the same at your bank, for for mine "electronic statement" means a PDF... which I can easily store myself without the clutter of paper.
    – Andy
    Dec 7, 2019 at 19:03
4

Heterogenous archives

A big issue for historical data in banking is that they don't/can't reside within a single system.

Archives of typical bank will include dozen(s) of different archives made by different companies on different, incompatible systems. For example, see http://www.motherjones.com/files/images/big-bank-theory-chart-large.jpg as an illustration of bank mergers and acquisitions, and AFAIK that doesn't include many smaller deals. For any given account, it's 10-year history might be on some different system.

Often, when integrating such systems, a compromise is made - if bank A acquires bank B that has earlier acquired bank C, then if the acquisition of C was a few years ago, then you can skip integrating the archives of C in your online systems, keep them separate, and use them only when/if needed (and minimize that need by hefty fees).

Since the price list and services are supposed to be equal for everyone, then no matter how your accounts originated, if 10% of archives are an expensive enough problem to integrate, then it makes financial sense to restrict access to 100% of archives older than some arbitrary threshold.

Improve this answer
1
  • Good answer but it doesn't explain why they are unwilling to say "All records from now on". It shouldn't be too difficult to design something which allows the required data on the current/next system to be easily migrated/exposed to future versions.
    – Basic
    Oct 6, 2014 at 22:28
3

Well, I know why the Rabobank in the Netherlands does it. I can go back around one year and a half with my internet banking. But I can only go further back (upto 7 years) after contacting the bank and paying €5,- per transcript (one transcript holds around a month of activities).

I needed a year worth of transcripts for my taxes and had to cough up more than €50.

EDIT It seems they recently changed their policy in a way that you can request as many transcripts as you like for a maximum cost of €25,- so the trend to easier access is visible.

Improve this answer
2

If you need access to your data beyond the online availability, you download the transactions and manage the archive yourself. Six months to eighteen months is generally enough time for most people to manage their own archived data.

Big banks have the power to store and retrieve all the data online. Unfortunately, the older records are not frequently accessed. Why have these records online when they will be rarely accessed? Backing up data will take longer. Queries to retrieve data will take longer. Everything will take longer just so you can have records that 99% of customers will never access.

Improve this answer
0

Well, some banks are seeing the light. In their most recent redesign, Alliant Credit Union has an option to download all transactions:

Alliant Credit Union lets you download all transactions

Improve this answer

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .