“As of market close on October 16, 2020, Schwab made the decision to restrict access to its legacy OFX (ofx.schwab.com) for all third parties except for: Quicken (personal finance management), TurboTax, and H&R Block (for tax preparation). This step is being taken to secure our clients’ data; and we sincerely apologize for any inconvenience it may cause. Third-parties who have historically used OFX to aggregate Schwab client data (such as StockMarketEye) are encouraged to register for access to the [new] Retail Account Aggregation API at “developer.schwab.com”.”
Now you can't even manually download an OFX file from their site, only useless CSV files.
But there's an easy solution! Just read their FAQ: https://www.aboutschwab.com/customer-data-protection#sf237008166#sf239142570
And then, Robert, as a "third-party data aggregator", just sign up as a developer at
I don't use Schwab, so not missing anything... but the strange part (to me) is that they're claiming "protection", but will only provide access to "approved aggregator entities", stating that it must be an official/registered company and hence the ability to *formally* agree to their terms. They could have easily provided a means of generating OAUTH tokens out of band and supported OFX 2.x, with no concern for who's on the other side. I can see why they may want something like this for "middle man aggregators", but not end-user client software (?).
Robert, I had asked if you might be able to implement gauth, which you interpreted as oauth. This is what I had im mind as a possible solution: https://github.com/gbraad/gauth
I'm still hoping for some type of resolution from Schwab, in response to my letter to the CEO (they claim they're working on it).
Best I can tell... and this is the first I've looked... the gauth project is an open source OATH client that's intended to provide a time-sync'd password between user/resource for one time access (similar to RSA device tokens for remote login). I'm not sure what it would be used for w/ OFX?
I assume they already did force SSL. And if they generated for you random logins and passwords for OFX (e.g. read only) use that would eliminate having to give your web login and password to those "third parties".
So their Customer Service person told me to call Technical Support. After consulting offline "Bob" created a trouble ticket to see if my accounts (or my "PocketSense Application") could be put on the "exception list" with (surprise, surprise) Quicken. We'll see.
Sorry if this is the wrong place to pose this question...
But is anyone having problems downloading transactions from Fidelity NetBenefits? It was working a week ago and this morning i was forced to change my password. Now, every time i try to download it disables my password. When i look at the logfile I see error code 15500. Any suggestions?
Sam A, if you are using PocketSense, this is the right place.
I am using Fidelity (not NetBenefits), and PocketSense download worked this morning triggered by my scheduler. I just tried a Test in case something had changed, and that worked too.
Do you by chance have a different computer somehow accessing this account?
Another possibility is that somebody entered the wrong password for your username manually, triggering the lockout. I think it is 3 consecutive fails before Fidelity locks you.
Cal - Thanks. Yes this was my error. I had two accounts with Fidelity and using SETUP I had updated only one of the accounts. After i updated both, it is working. What is interesting is that all it takes is for one incorrect attempt before the account is locked. THANKS AGAIN for the response and assist!!
Sam A: With the newer versions of PocketSense, PocketSense will not try multiple retries with the same username and password if you set self.skipFailedLogon true in site_cfg.py.
You can tell if your version has this feature by seeing if it contains "skipFailedLogon" in site_cfg.py, or just look at the date of the site_cfg.py file. Mine is dated 12/18/2019, but it was not the first to have the feature.
Then your sites.dat must contain this line to implement the feature: skipFailedLogon: Yes
That line is included in the new sites.template which is what you should base your sites.dat file on. When putting in a new version of PocketSense code, I merge the new stuff from the new sites.template with my old sites.dat. I suspect you may lack that line, and PocketSense tried logging in with your username and password for each of your accounts.
I just moved my Roth IRA, wife's Roth IRA, and a brokerage account from Schwab to TDAmeritrade (and yes, I understand the irony that they are both owned by Schwab). TDAmeritrade OFX transactions are working smoothly. It took about a week for the full transfer to occur (including liquidation of partial shares and transfer of residuals).
In Money, I just renamed the account, imported and voided the new transactions, and manually put in the sale of the small fractional shares. All of my history and everything is still up to date.
I've got a TDAmeritrade account that still works, but anticipate it will eventually be "migrated" to schwab.com. Probably just move my Schwab accounts to Fidelity if they continue on with their "customer centered" improvements.
I am trying to setup Checking and Savings Account with Bank of America and not having much success. The setting i am using is below. When i run setup, right after i enter user name and password i get an error saying "An error occurred requesting accounts from the site. Please check username and password. I have successfully logged in with the same username and password to ensure i have not been locked out.
I don't think BoA allows new users into the ofx servers, only those that were grandfathered in years ago. You can get qfx downloads from the webste though.
Having issues with Fidelity.... Has worked well for years, but this weekend GetData showed an SSL error when attempting to download... I called Fidelity web services, but their only suggestion was to change OFX.Fidelity to OLTX.Fidelity.... That didn't solve the SSL problem.... Anyone else having this issue ???
Ernie K: I downloaded from Fidelity, using a scheduled task, around sunrise Saturday, and my scheduler does not attempt to download again until early Tuesday, where I pick up any Monday transactions and closing prices. My Saturday and Tuesday downloads worked fine.
I suspect that you may have been attempting to download during a maintainance window. I think those tend to be the small hours of Sunday, after midnight just following Saturday, ET.
Having same problem since 11/12. Doesn't appear to be a window log in timeframe. It's all the time and a SSL issue. Validation of OFX files have been issue for several days based on this fidelity log-- http://www.ofxhome.com/index.php/institution/history/449.
unknown: I am successfully running Python 2.7.12, and getting OFX from Fidelity. It may be from python.org instead of ActiveState. To get the version number, start Python with the "--version" or "-V" switch.
In Edge, try https://ofx.fidelity.com/ The purpose is to check that the security stuff is working for you.
Expect a lock icon in front of the URL, and a "Sorry. The page you requested cannot be found on our site. " message.
I registered with the Schwab developer site back in october, and tried to input my (static) IPV4 IP's for OFX access. The approval is still 'pending' months later.
Anyone have any luck with APPID or other such changes?
Re: Schwab > input my (static) IPV4 IP This just show how utterly insane Schwab's is doing in the name of improving "security". Shutting down legitimate access from their paying customers then on the side allowing backdoor by IP restriction. Once that cat is out of the bag, I am most certain that there will be available unprotected Proxy servers (in the allowed IP list) that one can connect to get access to OFX again.
I totally agree, but that is what was in the profile page for my developer account so I went for it. Since they never approved my request I suppose that is not an option anyways. I would guess it's reserved for big players like Intuit. I'm fine to jump through more hoops but they have given my zero hoops.
Note on Fidelity, in case everyone runs into this issue again. Two days out of nowhere I started getting the error: You don't have permission to access "http://ofx.fidelity.com/ftgw/OFX/clients/download" on this server.
No changes on my end. Same scripts, same Python, same password.
Looking back at the PocketSense comments from June, I found the following fix. Add ":443" to the Fidelity URL. My Fidelity URL now reads: url : https://ofx.fidelity.com:443/ftgw/OFX/clients/download
The fact that others ran into this issue back in June, and I just ran into it now leads me to think they may be doing a rolling wave of updates/changes across accounts.
Carlos, Thanks for reposting that. I ran into the same problem yesterday -- and I had also just installed on a new computer (to confuse things). I had not run across that URL change looking back. But I made the change and it corrected the problem!
Updates from Fidelity stopped working for me on Dec 5 also. I called Fidelity. They had no clue.
I had to apply your https://...:443 fix AND upgrade to the newest pocketsense.
We really need someone who is willing to actively manage downloading OFX and provide support. It would be worth it to me to pay $10 a year or something...
I see Microsoft EXCEL 365 has a premium template that provides financial institution updates. Has anyone tried that? Not that I want to pay an active subscription to 365...
Re: Schwab unprotected Proxy server (in the allowed IP list) - bskc.vasvavgrxvaq.pbz (rot13) https://requests.readthedocs.io/en/master/user/advanced/#proxies
Hey Robert, Totally unfair question, as your software is not involved but I don't have anywhere else to turn :( Last month, Money stopped importing transactions for most (though not all) of my accounts. The Money import does not report any issues. I've looked at the OFX files from my bank and I see no discernable format or control value changes. From RBC, my credit card transactions come in, but not my chequing account, however from BMO my credit card transactions don't come in (Only account there). I don't know if it's an issue of the first account failing? I've tried manipulating the order in the .ofx, but so far Money's deemed any of my modified files corrupt. Any ideas?
In sites.dat, there is a variable named combineOFX which defaults to Yes. Try changing it to No. This should let each account load individually to find the problem one.
Thanks Andrew, but I'm not actually using PocketSense for these accounts. I download the .ofx and import directly to Money. As I said, it's not really a fair question, I just don't know where else to turn for someone with Money smarts. Anyway, I've combed through previous working .ofx files with the now failing ones and I see no material difference, and obviously Money hasn't changed, so I'm stumped. Might just have to put a bullet in it and see what else might be out there... :(
Ron Smith: To have somebody take a look at your OFX or QFX file, I suggest
1. Make a sanitized version of the file. Use Notepad or other editor that will not add formatting characters. 1a. Replace your account number content with something like 12345678. 1b. Replace occurance of your name in transactions, if any, with ABCDEFG or some such. 1c. Do not make changes in any characters that look like punctuation or math symbols. Avoid changing amounts, because some of the amounts have a required relationship, such as field 1 plus field 2 always equals field 3.
2. Put the sanitized file into a zip file.
3. Put that zip file somewhere accessible. I suggest tinyupload.com The process will give you a URL for public access to that file. Copy that URL.
4. Paste that URL into a post. Somebody will probably tell you what keeps your OFX file from being accepted by Money.
Note that https://social.microsoft.com/Forums/en-US/home?forum=money still exists for read-only access. https://groups.google.com/g/microsoft-money is a non-PocketSense Microsoft Money group.
Hi Ron, You can also use Microsoft's OFX Analyzer to see any errors in the file. I find it useful to see why a file doesn't work (or why some transactions don't show up--that's usually due to the price*quantity not being equal to the total). https://microsoftmoneyoffline.wordpress.com/ofx-file-analzer/
Andrew, thanks for the tip on OFX Analyzer. I checked it out. It has no complaints about the files in question. Here's one example:
***OFX 2.0 Add ~2 Running Data Through Parser No Parse Errors
Reading Data Into Buffer Tokenizing Data Initializing Data Structures Analyzing Data Verifying Security Types Verifying Security Uniqueness Verifying FITID Uniqueness Verifying Sign Correctness Verifying Total Calculations Verifying Other Done Analyzing File
Cal, just want to confirm before I anonymize and upload: to your comment "what keeps your OFX file from being accepted by Money." - Money does not complain about the files. I get "Import complete" every time. Money even updates the "Bank balance" correctly - I just get no transactions. Let me know if you think the exercise is still worthwhile
Since Money doesn't seem to be complaining about the imported file(s) could it be that you have inadvertently applied one of the filters? e.g. Unaccepted Transactions or Unreconciled Transactions
Make sure that the All Dates and All Transactions options are selected from the 'View' drop-down list for the registers in question.
Regarding Money Plus Sunset, you might installing that on a different computer to check to see if there are issues. Money Plus fixed Spinoff, Backup to USB flash drive fixed with >2GB free, it honors number of backups setting for scheduled backup, and probably more.
I don't know if there would be issues with Canadian files. I am thinking Money 2006 for Canada may have only used "Essential" reports and other thing, and not "Advanced". I am not sure.
Just to close this thread off, I also posted in the Google group. As I result I tried Money's "repair money file" feature, which seemed like it may have done the trick, but not completely, i turned out :( . As of now, I've started a fresh .mny file. So far, so good.
Ron, because you aren't using Pocketsense, you aren't getting the benefits of "scrubbing" the ofx files. So look for those items, for instance with Schwab...
Pocketsense scrubber for schwab: # 17-Feb-2013*rlc # - Added scrub routine for CORRECTACTION and CORRECTFITID tags (not supported by Money)see if those are in the problem files.
Thanks Ameridan. It's been some 14 years, so recall is a bit sketchy, but I don't think (?) the Canadian banks expose the endpoints pocketSense would need. I'm game to try, if anyone has a sample for RBC or BMO. In any event, Money does not complain about the files. When I monkey around with them to test out some theories, Money is pretty consistent in complaining the (modified) file is corrupt, but using the files as provided by the bank , Money reports "import completed" every time.
Amex card transactions download stopped working since yesterday. https://online.americanexpress.com/myca/ofxdl/desktop/desktopDownload.do?request_type=nl_ofxdownload responds with a timeout or "Internal server error".
It was totally my error. I had changed my username. I messed up in my method of making the change in my unencrypted ofx_config.cfg causing my changes to not be made.
I noticed that USAA just updated their OFX servlet location as of 1/25/2021. I can't find the latest URL anywhere. Would someone be kind enough to post the URL please?
MoneyDance shows the url as https://service2.usaa.com/ofx/OFXServlet I don't use USAA, so I am unable to tell if the is the old or new one. You can always see what they use at http://moneydance.com/synch/moneydance/fi2004.dict
MoneyDance users are discussing this at https://infinitekind.tenderapp.com/discussions/online-banking/17962-cant-download-usaa-transactions Post #12 says USAA has a new direct connect procedure-- (USAA Federal Savings Bank-New) is the new institution to setup and it requires a UserID and Pin assigned by them-not your old USAA member number. This was changed as of yesterday.
Yeah, unfortunately, for Pocketsense users, none of the information will help. Since I use Microsoft Money, I have to manually configure PocketSense for the updated ofx server. It seems that some folks over at MoneyDance are still having issues after following the suggestions from the tech support team. Hopefully, someone posts the updated bank information.
In the meantime, I'll check out that MoneyDance page.
So, it looks like USAA is going to a new system where the application needs to first authenticate with their servers to request application access as shown in this page: https://infinitekind.tenderapp.com/discussions/online-banking/17962-cant-download-usaa-transactions
Apparently even MoneyDance users are having problems with the site now. I'm wondering if PocketSense will be able to work with a site that requires this new type of authentication method.
One more update...I discovered that USAA moved the ofx server to the following URL: https://df3cx-services.1fsapi.com/casm/usaa/access.ofx This is according to this page: https://lists.gnucash.org/pipermail/gnucash-devel/2021-January/045665.html Not sure if that's enough for PocketSense to be able to reconnect to USAA but I would be more than happy to throw in some coffee change if you can get it fixed!
More information is coming to light. It would appear that USAA has switched to OAuth using Intuit's servers in the transaction process. The developers of Gnucash and Moneydance are both trying to reverse engineer what is going on behind the scenes. If this is indeed the case, will Pocketsense work? The software would need to first connect to the Intuit servers and then request a password and pin through the software. USAA would then add the software to the security profile on the account.
Just checked OFX Home status for USAA FSB (https://www.ofxhome.com/index.php/institution/view/483). Though that web page suggests the published OFX link information is "Validated", a quick peek into "View History" shows the following: ... SSL passed validation on January 26th, 2021 OFX passed validation on January 26th, 2021 SSL passed validation on January 27th, 2021 OFX failed validation on January 27th, 2021 ...
The OFX status is "failed" ever since January 27th. I will join the throngs calling USAA next week to request some support.
I'm hoping that Robert can chime in and see if he can take a look and see what's going on. It would appear that sending angry messages to USAA isn't cutting it. By the way Ron, I see a client_id in the URL above. Was that copied from Quicken? I would imagine we would need to generate a unique id if we try to request an app PIN and password.
That URL was copied from another site. And you may be correct, but you need to navigate to the URL only after you've logged into USAA. And after doing that, Quicken showed up as a connected app on the Security page of my Profile in USAA.
I have noticed something else that may be related to that URL ID. The documentation states that if you lose your id/pw, you can get another. However, if I disable the "connected app" in USAA, and then use that link to get "another", the returned ID/PW is identical. So that client_id may be Quicken generated and have something to do with that. I also noticed, this morning, that the Quicken as a connected app in my profile has disappeared! With all the back and forth testing, I'm not sure if that was something I did or something they did. But since the URL always seems to generate the same ID/PW combination, I was going to disconnect it anyway, on the off chance that someone would get the same combo as me using that link.
on, and Capt_Ahab: I am interested your observations, mainly in that this could propogate to other FIs. I fear this ID is somehow signed by Quicken, and even if you could figure out the protocol, that ID would stop working when your Quicken subscription expired. I may be overly suspicious. Can you create another Quicken file for a different identity, such as a spouse? I suspect that will generate a different ID despite using the same URL. Cal Learner
I myself did the same thing after testing to see if I could at least "trick" USAA into thinking that I was registering the Quicken app in hopes of using the resulting id and pin for OFX authentication but unfortunately it's not working. Robert's latest post below on Feb 2nd seems to indicate that this issue is beyond the scope of PocketSense. I am keeping an eye on the Moneydance tech support forum since that product also relies on the OFX standard to connect to financial institutions.
I've also been monitoring the USAA Community forum and there is mention of this FDX standard that is a coalition of companies to include Intuit and USAA. I don't know if this is a replacement for OFX or what but here is a link to the page showing USAA as a member: https://financialdataexchange.org/FDX/The%20Consortium/FDX/The-Consortium/Members.aspx?hkey=362ecd23-b752-48aa-b104-a99e916276c8
I don't have much to add re the USAA issue, except that the OFX request/response schema used by the scripts doesn't support OAUTH. The scripts are based on the OFX 1.x Standard, where authentication is performed via user/pw, plus a unique client_uid provided by the bank on first connection when ofxVer=103. Technically, there would be little difference for subsequent requests if the bank instead provided the access token using a separate / out-of-band connection. I suspect that something like this may be happening w/ Quicken, where the token is being assigned independent of ofx requests.
According to a commenter today on the MoneyDance blog: "According to support, their new authentication tokens have to be built into the software". I wonder if that is something that could be added to the PocketSense scripts. Of course, I have no idea exactly what USAA means by that nor how it could be accomplished. Maybe more information will become available in time.
I don't have USAA but a Moneydance user sent me an email and mentioned that manually CSV download is available (as current work-around) and if I can help him convert the CSV into OFX. I have some codes that I was able to re-use and create a command-line tool for him. If this is something that you can use, see http://bit.ly/3cPajDZ
I already have a CSV=>OFX converter, but the entire process -- Log in to USAA Web interface; select account: select the date range to be downloaded; download; convert; open with MSMoney; check the transaction matching for four accounts (two of which are pretty active) takes a heck of a lot longer than merely opening PS and running the script. Not a good long-term solution for me as it takes much too much time and personal intervention. I have applied for a Fidelity credit card (where I already have brokerage and retirement accounts). Once that is approved, if USAA remains recalcitrant, I will be moving my banking business.
Ron said "I already have a CSV=>OFX converter, but the entire process -- Log in to USAA Web interface; select account: select the date range to be downloaded; download; convert; open with MSMoney; check the transaction matching for four accounts (two of which are pretty active) takes a heck of a lot longer than merely opening PS and running the script."
Ron, a csv or qif to OFX converter must produce a transation id. Besides being unique for the transaction, the same FITID should be produced if the same transaction is re-downloaded. One way to do that is to include date, amount, and payee in the generation. In addition, the download should be checked for two identical transactions, and generate two different FITIDs if that happens.
The reason I bring that up is that if you are spending time matching transactions, that may be because your converter is not doing a good job generating FITIDs.
My current bank, Citibank, lets me download OFX by web, but it uses a different FITID each time the same transaction is downloaded. I don't have a lot of transactions at that bank. I have a Citibank credit card with them, and their OFX works flawlessy with PocketSense.
I have the Fidelity credit card too, and it works well. PocketSense cannot download the transactions, but does download the payments, which I convert to transfers to the CC. The CC transactions can be downloaded via web. The FITIDs are good. The payments match the already-there transfers in Money.
I hope that USAA is willing and able to fix this. I agree with those that say that the change was probaby motivated by Quicken. I suspect it was Quicken that provides the OFX/QFX server software, and maybe more. So the new IDs would serve as keys. I also suspect the IDs would stop working when your Quicken subscripion expired.
It seems like someone finally cracked the code to at least get gnucash to interface correctly with USAA. Can someone get PocketSense to adapt to the new method? Here is the link to what they discovered: https://lists.gnucash.org/pipermail/gnucash-devel/2021-February/045690.html
Hey Robert, are you the one who maintains the PocketSense script or is it someone else? It looks like we have the fix but the script would need to be programmed to change the way that it sends the request to the OFX server. I'm guessing this would be accomplished in the ofx.py module.
Call USAA Customer Service (210-456-8040) and ask for the "IT Department." IT is very knowledgeable as to what is required to comply with new changes (made without notice), and familiar with "Microsoft Money." Thanks
USAA Bank handling development idea: I, or somebody else who can generate Python code, but is not a USAA bank user, could help get this USAA thing going. I am pleased to read that the USAA tech people are supportive of this. I was afraid that would not be the case.
1. How would I get a request OFX into a file as PocketSense will send it to the FI? I know that I should be able to figure this out, and I probably could eventually, but somebody probably already knows.
2. What is the current site descriptor for USAA bank?
3. Given those, I would propose to post a USA request file to tinyupload.com, with pretend account number and password. Then I would propose that somebody modify that with Notepad etc to what they think would be the request file that meets the new criteria.
4. I would modify one or more of the *.py files to make the modifications. The modifications would not be as elegant as the rest of the code; it would be less object-oriented and more Kernigan and Richie style. It might involve a request scrubber, or be done some other way.
5. Some user would inspect the modifications to see there was nothing nafarious in my modifcation, and try it out.
6. If that code works, at some point, Robert would probably incorporate the features in a more readable style. That could involve adding one or more possible entries to the sites.dat descriptors.
I did some experimenting. I put 3 files into a zip file: ofx.py (modified to write the bank request to a file for debugging) reqq (the request file I got by logging) usaanote.txt (notes)
SHA-1: 533c7d5d80e1d0cac51d0e1cb600b700d5dbd789 usaa.zip If SHA-1 doesn't interest you, then ignore it.
File was uploaded successfuly on TinyUpload.com server. You can download it from address: http://s000.tinyupload.com/?file_id=45164446686448891753
There is a program ofxTools (https://github.com/csingley/ofxtools) that has recently been updated to use the new USAA protocol. There is also a thread regarding Gnu-Cash (https://lists.gnucash.org/pipermail/gnucash-devel/2021-February/045704.html] which discusses the new `site.dat` entries for the new USAA protocol. These might be useful to someone knowledgeable. Unfortunately, ofxTools requires Python3.6 or 3.7+, and PocketSense won't run under Python3+.
Yeah, the other issue with ofxtools is that it doesn't appear to be as user friendly as PocketSense. It doesn't have a menu driven setup and you basically have to give it parameters. I suppose you could create a batch file to run it with the needed parameters but I'm not sure how easy it would be overall. Ideally, I'm hoping that Rob would be able to look at the notes from the gnucash forum and see if he can implement those changes in his script. Hopefully this won't require too much time to implement.
Just a quick update....One of the CSRs at USAA has posted a link that will let you generate your own clientID, Access ID, and Access PIN. You have to log into USAA first and then open this link: https://www.usaa.com/accessid After accepting, you will see the clientID in the URL on the page with the ID and PIN. I would copy that and save that with the other important information.
Hi Rob, I was wondering if you have plans to modify PocketSense to work again with USAA. At this point, I think we have figured out how to grab the necessary information (clientID, AcceessID, and PIN) so that removes a huge issue. The PocketSense software would just need to be able to send the request to the new USAA server in the format as specified in the gnucash forums (as Ron has linked). You're basically changing the format a bit and sending the clientid value in the script. Please let me know if this is workable or not. Thanks!
I don't use USAA so no way to test, and am unable to focus on it at the moment. However... if someone can write out explicit rules required for USAA ofx connections then I'll see what's needed to fold into the scripts. By "rules", I mean exact differences vs standard OFX requests. Assuming it's still OFX 1.x, then I suspect something very simple like "use this code for username, this one for pw, and this one for clientUID". Of particular importance is whether clientUID is server-assigned or must be entered by the user. The scripts currently only support auto-assigning the clientUID on first connect w/ the server. Keep in mind, though, that changes that are too specific may break other connections, but we'll see. - Robert
Hi jmf, I am familiar with this process but have you had success getting PocketSense to download transactions?
Rob, I'll try to copy and paste some of the pertinent information from the gnucash forums where they go into detail on the format of the ofx request. This also includes updating the USAA ofx server URL.
Here is some good technical info from the gnucash forum discussion that goes into the ofx request format:
I got this working in my software with some help for the info on this list. Here is a write-up:
USAA's changes to their OFX interface -------------------------------------
On 2020-01-26, USAA's previous OFX interface ( https://service2.usaa.com/ofx/OFXServlet) stopped working. It seems like they switched to a new interface through a tech provider to replace their previous login method (with your website credentials) to an app-specific ID and password. This is a good move for security, but it was done without notice, it seems, to anyone but Quicken.
>From some internet searches, I found some people on the right track to fixing this on the GNU Cash development mailing list:
They were able to determine that USAA was: - using a new OFX endpoint: https://df3cx-services.1fsapi.com/casm/usaa/access.ofx - using a new OFX Org ID: USAA Federal Savings Bank - using a new OFX FID: 67811
Additionally, someone on the USAA forums was about to extract and post the link to generate an App ID and PIN:
However, with a lot of trial and error I still wasn't able to hit this new endpoint successfully. So I decided to give the devil his due and temporarily got a Quicken subscription and setup an SSL man-in-the-middle.
The new OFX interface is *very* finicky, so you basically have to input everything exactly the way it expects it. Here is an example of an account listing query that works:
echo -en "OFXHEADER:100\r\nDATA:OFXSGML\r\nVERSION:103\r\nSECURITY:NONE\r\nENCODING:USASCII\r\nCHARSET:NONE\r\nCOMPRESSION:NONE\r\nOLDFILEUID:NONE\r\nNEWFILEUID:NONE\r\n\r\n\r\n\r\n\r\n20210207031942\r\nXXXXX\r\nNNNNN\r\nENG\r\n\r\nUSAA Federal Savings Bank\r\n67811\r\n\r\nQMOFX\r\n2300\r\n1955A543-B071-455E-A31E-73CC7C493D68\r\n\r\n\r\n\r\n\r\ne39add7d-2085-4504-b9ee-be37927de39c\r\n\r\n19900101\r\n\r\n\r\n\r\n\r\n" | curl -isS -X POST -H "Content-Type: application/x-ofx" -A InetClntApp/3.0 --data-binary @- https://df3cx-services.1fsapi.com/casm/usaa/access.ofx
Note you have to change the XXXXX and NNNNN to the App ID and PIN you get from the link above.
Some things I've found through trial and error: - The OFX elements must be separated with "\r\n". This is dumb, but true. No spaces. No simple "\n". Exactly "\r\n". - The APPID "QMOFX" and APPVER "QMOFX" work. Others I tried did not. - The CLIENTUID "1955A543-B071-455E-A31E-73CC7C493D68" works for me. It must be uppercase. This might be particular to your account. If so, you can find it looking at the OFX logs from Quicken. - TRNUID must be present, but an UUID will do. - DTACCTUP: The value "19900101" works. The value "19700101" does not. The value "19900101000000" does not. - You need the "Content-Type: application/x-ofx" header - You need the User-Agent "InetClntApp/3.0". This is what Quicken for Mac sends.
It also seems their gateway will under some conditions put your IP on a ban list. If you are testing, you may want to spin up an AWS instance or something.
Thanks to Bob White on the GNU Cash list and RDD! on the USAA Forums for the breadcrumbs. No thanks to USAA for swapping out their functional interface with absolutely no notice or documentation and pretending like Quicken users are the only customers of any importance. Please just don't break our software again... at least for awhile.
The only change is that it appears that the clientID can be found within the URL after going to https://www.usaa.com/accessid and going to the page that issues the AccessID and PIN. This could be an additional value that you can add to the sites.dat file that will need to be present if the user is trying to get transactions through USAA.
Scott McRae: If you want to show OFX/ SGML in a post, you can change all < characters in your post to < Similarly change all > characters in your post to these 4 characters: >
Did you look at my February 10, 2021 at 4:49 PM post?
I skimmed over the link above (https://lists.gnucash.org/pipermail/gnucash-devel/2021-February/045704.html) and summarized the following. Please review / commment.
1. They appear to have a tool where you login and generate an appID and token (PIN), which would be used as the userID and userPW. 2. OFX elements must be separated with "\r\n". 4. DTACCTUP must be yyyymmdd format.
*ClientUID*: It's not clear whether clientUID is assigned during connection or pre-assigned by the bank? If someone can use their online tool to see what it gives you, it answers the question. If they only give the appID and PIN then the clientUID is created during initial connection, per OFX 103.
The ClientUID is contained in the URL when you create your access ID/PIN. After enrollment, it directs you to a webpage with your ID/PIN. The URL had the following form for me:
I have managed to get a successful connection using edited pocketsense scripts and clientuid, access id and access password from USAA. I'm not getting account info returned because I believe I've tried and failed too many times via trial and error from same IP. I'm going to try what I have from another IP address.
Hi John, when running the PocketSense script, are you putting the clientuid into the connect.key file? For me, I have two entries in there so I'm unsure which one to edit if this where you are making the change.
Use setup.py to find the clientUID (connectKey). The file is partially encrypted on purpose, but not the key itself. Use 2. List Accounts, and Y to "show connection keys". You can then change that one in connect.key to whatever you want. The connectKey is created once for an account connection, and would only be regenerated if the account is deleted in setup and then recreated.
p.s. Note that the key is between single ' quotes. Don't delete those. Also, I'd make a copy of the file before editing, and only use something like Notepad++ that doesn't add hidden CR/LF chars.
One other question...Are you using all caps for the clientID? Also, I tried to set the username and password again through PocketSense and it's not taking the new AccessID and PIN. Is that what you're using for username and password?
I added an additional field to the USAA site in the sites.dat file which contains my clientUID. This value was obtained from the URL where you get your new access id and pin. This is in the URL prior to clicking the Allow button. What Andrew is pointing to above is not the client did. When this gets added to the connection string it must be in all caps. My code mod converts the data in the sites.dat field to uppercase for the clientUID field. I send this as the clientUID rather than the one the scripts calculate.
One more question John, where did you get the edited version of PocketSense that you are using? I think that's my issue because when I try to change the username and password to my AccessID and PIN, the script informs me that there is an issue with the username and password.
Ahh...would you mind sharing your code mod then? I am using the standard PocketSense script at the moment. Also, could you provide an example how you insert the clientUID field into the sites.dat file? I simply overwrote the information between quotes for the exiting one generated by PocketSense. Also, what are you using for username and password. Is this this now the AccessID and PIN or just the normal USAA username and password we were using before the change?
I'm modifying the Pocketsense code on my home computer myself. The current released code isn't going to support the changes to the USAA site. I'm a software analyst/developer with background in c and c++. I'm picking up Python from looking at the scripts.
Ahab - Sorry, I got tied up at work and didn't get a chance to post my changes. Here is a brief synopsis of what needs to be done to set up a connection to the new USAA OFX server. 1) Log into www.usaa.com using your normal username and password combo 2) Navigate to www.usaa.com/accessid to enable Quicken access and get your new access ID and PIN 3) The URL of this page contains your unique client_id, which you will need to include in the sites.dat file 4) Once you provide access permissions and click the "Allow" button you will be taken to another page that identifies your access ID and PIN. These two values need to be supplied as the username and password for access to USAA through the Pocketsense scripts. You will be queried for these when you attempt to set up an account the first time for the new USAA. 5) Create a new site definition for USAA using the info supplied in one of the messages above. Add a clientId field to this site definition that contains the client id that you obtained in step 3. 6) The ofx.py, Setup.py, and site_cfg.py files have to be modified in order to get them to format the OFX string to be sent to USAA correctly. The site_cfg.py file needs to be updated to read a new clientId field from the USAA site definition. The Setup.py file needs to be updated to create a different DTACCTUP value for just the new USAA account. This value should be 19900101. The value used by other accounts is 19700101000000. The ofx.py file needs to be updated to use the clientId for USAA rather than the clientId from the connect.key file. This file also needs to be updated to insert 'User-Agent', 'InetClntApp/3.0' rather than 'User-Agent', 'PocketSense' in the OFX header. I suspect there might need to be some changes to GetData.py but I'm stuck at this point because I can't get account info returned due to having tried and failed too many times to get the correct string sent over. 7) Get account info downloaded from USAA and into the Pocketsense database. 8) Try to retrieve transaction data from USAA using the updated scripts (might require some additional updates) and downloaded accounts.
My IP is whitelisted (i.e. blocked) by the third-party site that's using the Imperva Incapsula cloud management software. I verified this today with the Imperva operations center. I have to somehow get USAA to remove my IP address from the blocked list (easier said than done given that most CSRs at USAA don't know what whitelisting is or even what the Incapsula software is).
After looking at the details of what needs to be done, I think I will just need to work with Rob and hope that he can decipher what changes seem to have worked for the folks at GNUcash. If you need me to try out a script change for USAA Rob, let me know and I can run and let you know what happens.
To all of the above: my humble thanks for your efforts to straighten out the mess that the corporate idiots running USAA these days created. I love using PocketSense but need step-by-step instructions for a compleat idiot such as myself on what/which/where/how to edit files in order for it to resume functionality. Why the powers that be at USAA chose to give Quicken a monopoly on their ofx facility is anybody's guess.
Today I was able to get an access ID and PIN from https://df3cx-services.1fsapi.com/casm/usaa/enroll, so we know that is no longer an obstacle. I do need somebody knowledgeable to spell out the rest of the fix, i.e., which files to edit and explicitly how, with step by step instruction from someone who will take pity on me. I'm sure Robert will be might grateful, too.
Respond to this and I'll tell you specifically what changes need to be made to the scripts. Besides the access id and PIN, you also need the client id from the USAA site.
P.S. When you get an access ID and PIN from USAA's link you'll get an acknowledgment email: "Quicken has been connected to your USAA account. To manage what apps are connected to your account, view your USAA Profile."
Your profile in USAA shows that you are allowing Quicken access your name, home address, phone #'s, email address, all account numbers, transactions and balances. I'm not too sure about that and may need to take my banking elsewhere, as much of a pain in the neck as it might be to switch.
I believe the info provided thus far is enough to add support in the default scripts w/o being specific to USAA. Note, however, that *all* OFX connections provide access to personal info you've listed. Also, it appears that when you generate the ID and PIN, you must copy the clientUID in the address bar at that instant. That key must match the ID and PIN on future connects. I can't test, but that's my take.
Also, for the partially paranoid such as myself I just re-discovered that there are check boxes on the ID and PIN "application" page:
Allow Quicken to access your USAA information Quicken is requesting permission to access your: []Account Owner Information []Bank Account Numbers and Details
I left the Account Owner Information box unchecked and will attempt to use the ID and PIN set provided and determine whether connectivity can be achieved without it. I see no need to volunteer my name, home address, phone #'s, etc. to Quicken unless absolutely necessary.
P.S. Please disregard what I just wrote above about leaving a box unchecked; I thought it worked initially but on multiple re-test attempts it errors out and will not provide ID or PIN numbers unless both are checked.
Hi Robert, John is right...Once you are able to get the clientUID from the site, from everything I've read, this value is permanent (as well as the AccessID and PIN) for all future connections.
I've updated the scripts to support a few extra site parameters, driven mostly by the USAA request, and a DEV version is available for testing. Use the site entry below, including your clientUID value. User-name (AccessID) and password (PIN) must be defined in the account entry via Setup.py (add new account). I also included some updates from a long standing "to do" list, which I'll explain when posting a final version. - Robert
Hi Robert! Thank you for getting PocketSense to work with USAA. I did run setup.py and selected to create a new USAA account and edited the sites.dat with the specified information above. I tried using an all caps clientUID as well as all lower-case clientUID. I keep seeing the following message now: "An error occurred requesting accounts from the site. Please check username and password." I'm not sure exactly what is causing the issue however. Would you happen to know if there is someplace I could check to see what is going on? I did make sure to use the AccessID for the username and the PIN for the password also.
Of course, I'm not privy to what's been done up to this point that may have caused an IP to get blocked. If that's the case, then it would have to be unblocked by someone on their side, unless there's a timeout of some type. Another option to consider, and won't hurt if you're already blocked, is to manually enter the account # rather than depending on them providing a listing during setup. That feature is separate from OFX requests, and may not be supported on their new server. Basically, answer "Y" to continue adding an account and enter manually... and test when prompted.
You can't open the ofx endpoint in a browser. That's prohibited by the rules for the site. If your IP was blocked you'd be getting the robots response that I am getting.
Ahab - Which page are you getting the client id from? It should be the page that has the checkboxes and Allow button, not the one that has the access id and PIN displayed.
Okay, I made sure to grab the clientUID from the screen where it has the checkboxes and the allow button. I put this in the sites.dat file and went to set up a new USAA account. This didn't seem to do anything. Can anyone confirm if they managed to get the DEV version of PocketSense working with USAA?
- I downloaded the new DEV and installed it over a copy of my functioning PS script - I edited sites.DAT as shown by Robert:
SiteName: USAA NEW url: https://df3cx-services.1fsapi.com/casm/usaa/access.ofx fiorg: USAA Federal Savings Bank fid: 67811 bankid: 314074269 appid: QMOFX appver: 2300 ofxVer: 103 dtacctup: 19900101 userAgent: InetClntApp/3.0 clientUID: my ID with all CAPS
I then attempted to set up automatically:
--------------- Configure account for USAA NEW
User name : xxxxx Account password: xxxxx An error occurred requesting accounts from the site. Please check username and password.
Continue configuring account (Yes/No): [N] y
***************** NOTE: If the account number is masked (e.g., XXXX-XX-1234), you MUST manually enter the account number *****************
Enter line #, *or* the actual account number
Account List ------------
Account # : xxxxx Replacing xxxxx for USAA NEW Do you want to test transaction downloads for the new account now (y/n)? y USAA NEW : xxxxx : Getting records since: 20210116 local variable 'query' referenced before assignment An online error occurred while testing the new account. -------------------
Here's the result I get when proceeding with manual setup and run the test at the end: Test account #: [0] 1 Test USAA | XXXXXXXXXX (Y/N)? y USAA : XXXXXXXXXX : Getting records since: 20210116 local variable 'query' referenced before assignment An online error occurred while testing the new account.
I suspect the issue is that the site parameters I listed above are only the fields I knew. I omitted account type, since I didn't know what it would be. For example, for a bank account:
Hi Robert, I did add AcctType to BASTMT and that error message went away. I then received some kind of OFX error message that states the following: Request unsuccessful. Incapsula incident ID: 1246010460002156800-3480998060623239.
One other note...I did have to manually enter the USAA checking account number as it still did not take the username and password for auto-setup. I'm not sure if that's supposed to happen or not.
Okay quick update....It works! The issue was that you MUST use the ID given on the page WITH the Quicken AccessID and PIN. If you use the clientID on the previous page, it doesn't work. I first tested with that java ofx tool first and then confirmed it's working with PocketSense as well.
Yeah, that's the one but I really like PocketSense better because you can just run the script to grab the bank info after it's been set up. Since hleofxquote is GUI based, I don't know if there is a way to use it in a startup script for Microsoft Money. I don't know enough about it though to say for sure.
By the way, I'm sending in a donation. If any one else wants to thank Robert for his hard work for fixing the USAA issue, please consider sending him some money. (Link here: https://sites.google.com/site/pocketsense/aboutme-1?authuser=0 )
Ahab: Tried setting up and I'm getting the "robot" response as an invalid OFX. (also has the `Request unsuccessful. Incapsula incident ID` line in the returned text) For clientID I used the `code=` segment of the URL on the same page as myu AccessID and Access PIN. I did not upperCase it. Did you? I did add `AcctType : BASTMT #bank` line to the sites.dat entry
According to Fleming, that response means my IP is being blocked. I think there is a time out for that, but I don't know what it is. Any suggestions appreciated.
Yes, I would use all uppercase characters for the clientID as this is what I used. It sounds like you have the correct one if you copied the 'code=' segment all the way up to the '&' in the URL. As far as the IP being blocked, I would just try it again and if it doesn't work, wait an hour or so and try one more time. If everything is working correctly, PocketSense will prompt you to select your account number after putting in the AccessID and PIN.
Sadly, it is still not working, in that I get the "Robot" response. I even came in from a different IP address (using a VPN) and got the same response. So my problem is not IP blocking. I don't have more time today to work on this, maybe later in the week.
Ron - The Pocketsense scripts attempt to send a given request to the OFX server up to three separate times. I believe the one that is giving the "Request unsuccessful" response is actually from the third attempt. This one is going to fail every time. Success has to occur on one of the first two tries to get a successful response. If you look at the responses from the first two requests I believe you'll just see an empty robots header. This empty header is the indication that your IP is blocked.
John, That may be. But since I started a series from a fresh IP address via a VPN, that does seem to point to a problem in my setup, and not IP blocking as the cause of my problem. Especially since Capt_Ahab, and you, both seem to have things working.
Ron: It sounds like something isn't quite right w/ your settings. I'd double-check every entry, especially the UserID, PIN and clientUID values (I believe clientUID must be UCASE?). In fact, case often matters for OFX params, including the url itself, since it's included in the request header. Verify that you didn't copy bracketing symbols from the url (i.e., leading/training chars). Grab the latest beta below for testing. If it still fails and you want to dig further, enable debug in control2.py by uncommenting the Debug=True line.
Ron - I can't get an account list nor download transactions, so I don't have things working. You might try swapping to debug mode and then posting your ofx query string with clientuid, username, and password redacted/removed. It would probably be best just to x them out, so we can see the string that's being sent over character for character.
One thing that I did yesterday that may have helped is that I revoked Quicken from my security profile in USAA and then went in and re-added it which does seem to generate a new clientUID. After this, I went and tried to add the USAA account in PocketSense using the newly assigned clientUID and then it worked. Please let me know if this works for you (or if you have already tried revoking and re-adding Quicken.
Capt_Ahab I didn't realize that a different clientUID was assigned. So this morning, Quicken had been automagically revoked from my permissions. I re-enabled it and used the NEW uid. But still, no joy. When it worked for you, what was the value for your appversion? I've seen some use QMOFX for both appid and appversion. Also, when it worked for you, were the account numbers automatically downloaded? or did you have to enter them manually after an initial error message?
I just now posted a new DEV version for testing. It's basically the same, but cleaned up a bit and made simpler. Specifically, I have deprecated the 3rd attempt for ofx downloads (leaving two), added userAgent='InetClntApp/3.0' on first call (previously omitted), and changed the default DTACCTUP value to 19900101. The userAgent and DTACCTUP values appear to be the current values used by Quicken, so I don't think it will break others. I'm leaving the optional site value for dtacctup and userAgent in case they're needed elsewhere.
If any of these changes cause issues w/ other sites, I'll revert to the previous logic. All sites/accounts that I personally use work w/ these settings.
Hi Robert, I tried this version but it seems to be generating errors that I haven't seen before.
D:\Pocket Sense>Getdata.py Traceback (most recent call last): File "D:\Pocket Sense\Getdata.py", line 54, in import ofx, quotes, site_cfg, scrubber File "D:\Pocket Sense\ofx.py", line 74, in import getpass, scrubber, site_cfg, uuid File "D:\Pocket Sense\scrubber.py", line 66, in import site_cfg File "D:\Pocket Sense\site_cfg.py", line 51, in from rlib1 import * File "D:\Pocket Sense\rlib1.py", line 21, in from http.cookies import SimpleCookie ImportError: No module named http.cookies
Hi Robert, I discovered that this morning, I was unable to pull transactions yet again from USAA. I did attempt run the latest script you posted but after it failed, I restored the first DEV version you posted but now I'm getting that "Request unsuccessful. Incapsula incident ID: 1337000090018244641-47402149679268425" error.
The new version uses a package that's apparently not in the standard py distribution, so I'll need to manually implement that part or see if there's a standard (equivalent) package. When you say "first DEV version", I assume you had a local copy (?)... since the new DEV replaced the original in the link. Stay tuned on that part... but odd that the same script worked yesterday but not now, w/ presumably identical settings.
This morning I noticed that my Quicken permissions in my security profile are no longer listed. I don't know whether they would have disappeared with a successful log in through Pocket Sense. Perhaps the credentials are only good for a certain period of time if there has not been a successful connection?
I have a quick update...after I waited 10 minutes and then used the new ClientUID after running the first DEV script, it started working again. I don't know if the ClientUID expires after a while or if running the new DEV script sent something to the ofx server and caused my IP to be blocked temporarily but I will keep testing this first version over the next few days to see if the connection will stop working if I leave things alone. That should isolate what caused the issue this morning. I should have tested the old script first before overwriting the python scripts and running it but I'll post the results in a couples days if it continues to work.
USAA appears to be using Incapsula (Imperva) as a security gateway. My guess is that there's a delay between generating your clientUID and it being distributed/updated to the gateway server(s).
Yeah...By the way, after setting up the account again, I had waited about two hours or so before running Getdata.py. When I ran it, it failed again. I went back into Setup.py and sure enough, it was back to pulling invalid ofx files and failing even though I had successfully re-added the accounts earlier. I'm wondering if maybe a new ClientUID is somehow being issued over time and the script would need to pull the new ClientUID each time or something.... I wish I knew more about how this whole setup works.
I doubt that the clientUID "expires", or if it does, it would be much longer. It's basically a "back door OAUTH", where you're generating an ID, PIN and token out-of-band. Citi has a similar process, except the clientUID is generated via ofx during first connect, but the user has to be on their "allow access to app" page at that moment. Once established, it's persistent.
This does seem to be the case from what I can tell. After about an hour or so of just waiting (not revoking and re-adding Quicken access). The original DEV script seems to be working and pulling the account info. I'm a bit reticent to run the new BETA script again because my IP became blocked when I ran it.
Hi Robert, I was doing some reading on this hleofxquotes software that also pulls OFX data from sites. The instructions tell you to grab the clientID in the URL prior to accepting and then retrieving the AccessID and PIN. When I do this for Pocketsense, the script isn't able to pull data but it works just fine in hleofxquotes. At the moment, the IP seems to be blocked so I will have to play around with this script a bit later. So far, I can't say that the latest BETA is actually working but I just need to wait for the block to reset and try again later.
Just fyi, but the newest version is less likely to throw an ipsec flag, since it only retries one time (vs twice for the version you're using). I'm thinking that I'll suppress the backup attempt altogether, since I've restructured the request header and no longer needed.
Hi Robert, I downloaded the script from the "new BETA" link. As I am running setup.py again, it seems that it will intermittently not be able to download the ofx file. Sometimes it will work and other times it doesn't. I haven't been able to figure out what causes the script to fail to download the ofx files. Since I updated it, so far my IP isn't getting blocked so this is good at least.
Quick update - I downloaded the hleofxquotes java tool and installed it on my home computer. Using that tool and querying for account info with the access Id and PIN and clientUID from the URL of the page with the "Allow" button, I am able to retrieve account info. I'm looking at the content of the OFX requests now to determine the delta from what Pocketsense is sending.
John, just to clarify, are you using the original DEV version from Robert or the newer BETA version he just released today? For some reason, the latest BETA version seems to cause USAA to block my IP temporarily but the DEV version seems to work. I will have to try this again to verify for sure the BETA version from today is causing my IP to get blocked.
So...I copied the directory with the DEV version from the 15th of February and ran it again to see if it would work but now it's blocking my IP again. This was just after I ran it a few minutes prior. I'm wondering if maybe the USAA site blocks the IP if you send to many access requests in a short time. Rather than the script, maybe it's the frequency of making requests that causes the site to temporarily block your IP.
I was also able to use the hleofxquotes tool to download my account numbers and the transactions.
I used the client_id from the "Allow" page URL (I used lowercase and it worked). I used my ID and PIN from that generated page (which have never changed through all removals/reauths).
It was able to retrieve my account numbers correctly. Putting in one (and later all) of my account numbers (they must be 10 digits, pre-pad with 0's if you have a shorter number; hleofxquotes does this correctly already if you download your account numbers) correctly pulled all transactions.
Andrew - Can you posted the sanitized version of your OFX account request. I see a few deltas in what is being sent from Pocketsense, but nothing major.
Sure. Here's the HTTP header for both (account list request and transaction request), only difference was the content-length: ========================================= POST /casm/usaa/access.ofx HTTP/1.1 Content-Length: 667 Content-Type: application/x-ofx Host: df3cx-services.1fsapi.com Connection: Keep-Alive User-Agent: InetClntApp/3.0 Accept-Encoding: gzip,deflate
=====================================
Here's the sanitized, successful "list accounts" request:
The only things that jump out to me as being different from what Pocketsense is sending out is the DTCLIENT field and the DTACCTUP field. The TRNUID is going to be different from request to request, but the format is correct. The accept-encoding field is not present in the HTTP header that Pocketsense sends.
DTACCTUP is only used when listing accounts, and is included in the PS scripts when adding an account. DTCLIENT is different format, and includes milliseconds + timezone suffix. These aren't required by OFX, but possibly something that the gateway checks (?). Accept-encoding isn't required for OFX http headers, but perhaps (again) the gateway does?
Here's one other difference I've noted. The header length sent over by hleofxquotes has content length set to 575. The script sent over by Pocketsense has that value set to 608. I've modified my Pocketsense scripts to use the longer DTACCTUP and the DTCLIENT with addition of timezone info. I think the content values should be the same given that nothing else is different (i.e. username, password, clientuid, trnuid are all the same length. Can't figure out how the count for Pocketsense is 33 characters longer.
Here's the header from hleofxquotes tool: POST /casm/usaa/access.ofx HTTP/1.1 Content-Length: 575 <<<<<<<========== Content-Type: application/x-ofx Host: df3cx-services.1fsapi.com Connection: Keep-Alive User-Agent: InetClntApp/3.0 Accept-Encoding: gzip,deflate
Here's the header from Pocketsense: POST /casm/usaa/access.ofx HTTP/1.1 Content-Type: application/x-ofx User-Agent: InetClntApp/3.0 Host: df3cx-services.1fsapi.com Content-Length: 608 <<<<<<<========== Connection: Keep-Alive
I can't tell what's actually sent over. I'm surmising based on the content length that it's not sending both. I'm looking at the output file that gets printed by hleofxquotes showing the request that was sent. The eol marker is about the only thing that can be different as the rest of the data is identical.
I've noticed that Pocketsense is using TLS v1.2 protocol and hleOfxQuotes tool is using TLS v1.3. Don't know how much difference there would be due to updated encryption protocol, but perhaps there is something. The only two deltas I see in the data string I'm transmitting via Pocketsense and the one I'm transmitting with the java tool is the TRNUID and the DTCLIENT. Everything other piece of data is identical.
Sure thing! Here is the link to GitHub where the project site is located: https://github.com/hleofxquotes/hleofxquotes and here is a page where the developer put a link to a bitbucket URL with the version that fixes USAA: https://infinitekind.tenderapp.com/discussions/online-banking/18262-usaa-using-hleofxquotes-to-download-ofx Just look for the poster listed as hleofxquotes to find the post with the direct download link.
It looks like his main site for hleofxquotes doesn't seem to show a new release as of yet (maybe because it's in beta at the moment)
I have installed the new beta this morning, including the new sites.dat (*below), and after entering my ID and Password Setup.py disappears without a trace. Here are the error messages when I run **Getdata. Please advise, thanks.
** invalid literal for int() with base 10" ': 103' .\xfr\USAACHECKING...94.ofx is not recognized as an internal or external command..."
(Note that I didn't add all the numbers for security reasons)
Also, what information should I use when Setup.py asks for Name(ID) and Password? Previously I would enter the same credentials I used to sign in to the usaa web page. Thanks
Not sure which version you're using, but I just now uploaded another DEV version for testing. It's similiar to the last beta I posted, but using \n separation rather than \r\n. I tested this version w/ the sites I use and it works correctly w/ those.
As for the Name / Password, from what I've gathered that's the ID and PIN assigned by USAA via the "enable connection" form. I don't use USAA, so just going by what others are posting.
Regarding the error above, that actually looks like you may have an invalid character in the site name that's throwing a file load error. Try renaming the site w/ no spaces or special chars (e.g., USAA-TEST would be fine). The scripts attempt to cleanup invalid chars from site names when creating files, but there's possibly a character in there that's missed in the filter but not allowed in windows file names.
I posted this as a reply above, but want to separate it as a new post. I uploaded another DEV version for testing. It's similar to the last beta I posted, but using \n line separation in the ofx-message rather than \r\n. I hasn't mattered for any other ofx service, but worth checking.
Hi Robert, I tried the DEV version and the test run on the account is still failing. Checking with the hleOfxQuotes tool, I can see that my IP isn't getting blocked at least.
I forgot to mention that the dev of hleOfxQuotes did provide some documentation on how the program communicates with USAA. Here is the link: https://bitbucket.org/hleofxquotesteam/hleofxquotes/wiki/USAA_Background_Notes
I've made it a little bit further in troubleshooting the USAA problem. I did an account request using the hleOfxQuotes tool and successfully received a list of accounts. I copied the DTCLIENT data and the TRNUID data from that request and inserted that data into the Pocketsense request and voila I finally received account data using Pocketsense. I suspect this means that the TRNUID is somehow derived from or tied to the DTCLIENT string.
Did you test with only the DTCLIENT field changed? TRNUID should be a unique value for each request, but clientUID is a static value across connections.
The DTCLIENT field has the following value: 20210217174516.169[-6:CST] The TRNUID field has the following value: f638e084-43a1-47d2-9a8d-ebe91441d664.
I'm using scripts that I've edited. These would probably be close to your latest dev version. I'm sending just a CR as a separator rather than a CRLF pair. I'm going to try to use just the time value from hleOfxQuotes tool with Pocketsense generated UUID for TRNUID and see if that fails.
Here's another piece of the puzzle. The TRNUID that's sent as part of an account query request is also echoed back in a successful response. Here's the TRNUID field in the request 314fb85b-f74d-405d-85bd-813f0c4472b1 and here's the TRNUID in the response 314fb85b-f74d-405d-85bd-813f0c4472b1.
To this point, the only successful response I've received to an account query request in Pocketsense is when I copied values for DTCLIENT and TRNUID from a successful hleOfxQuotes account query request and injected them into a Pocketsense request. Every other attempt has failed.
Well, now Schwab has gotten into the "unauthorized" OFX mode since the middle of October. Others have noticed and gotten some "answers":
ReplyDeletehttps://www.stockmarketeye.com/blog/schwab-brokerage-import-issues/
https://www.iggsoftware.com/support/articles/banktivity-6/schwab-is-offline-for-ofx-direct-connect/
The excuse is "customer protection":
“As of market close on October 16, 2020, Schwab made the decision to restrict access to its legacy OFX (ofx.schwab.com) for all third parties except for: Quicken (personal finance management), TurboTax, and H&R Block (for tax preparation). This step is being taken to secure our clients’ data; and we sincerely apologize for any inconvenience it may cause. Third-parties who have historically used OFX to aggregate Schwab client data (such as StockMarketEye) are encouraged to register for access to the [new] Retail Account Aggregation API at “developer.schwab.com”.”
Now you can't even manually download an OFX file from their site, only useless CSV files.
But there's an easy solution! Just read their FAQ: https://www.aboutschwab.com/customer-data-protection#sf237008166#sf239142570
And then, Robert, as a "third-party data aggregator", just sign up as a developer at
https://beta-developer.schwab.com/home
to use their special API (hint, it's not OFX).
I don't use Schwab, so not missing anything... but the strange part (to me) is that they're claiming "protection", but will only provide access to "approved aggregator entities", stating that it must be an official/registered company and hence the ability to *formally* agree to their terms. They could have easily provided a means of generating OAUTH tokens out of band and supported OFX 2.x, with no concern for who's on the other side. I can see why they may want something like this for "middle man aggregators", but not end-user client software (?).
DeleteRobert, I had asked if you might be able to implement gauth, which you interpreted as oauth. This is what I had im mind as a possible solution: https://github.com/gbraad/gauth
DeleteI'm still hoping for some type of resolution from Schwab, in response to my letter to the CEO (they claim they're working on it).
Best I can tell... and this is the first I've looked... the gauth project is an open source OATH client that's intended to provide a time-sync'd password between user/resource for one time access (similar to RSA device tokens for remote login). I'm not sure what it would be used for w/ OFX?
DeleteI assume they already did force SSL. And if they generated for you random logins and passwords for OFX (e.g. read only) use that would eliminate having to give your web login and password to those "third parties".
ReplyDeleteThey already have a setting in their Security Center to permit third-party OFX access, default off.
ReplyDeleteSo their Customer Service person told me to call Technical Support. After consulting offline "Bob" created a trouble ticket to see if my accounts (or my "PocketSense Application") could be put on the "exception list" with (surprise, surprise) Quicken. We'll see.
ReplyDeleteSince we are in control of appid and appver, Schwab could invent it's own "app" for us, but if they'll allow us in with QWIN 2800, fantastic!!
ReplyDelete-ameridan
Sorry if this is the wrong place to pose this question...
ReplyDeleteBut is anyone having problems downloading transactions from Fidelity NetBenefits? It was working a week ago and this morning i was forced to change my password. Now, every time i try to download it disables my password. When i look at the logfile I see error code 15500. Any suggestions?
Sam A, if you are using PocketSense, this is the right place.
ReplyDeleteI am using Fidelity (not NetBenefits), and PocketSense download worked this morning triggered by my scheduler. I just tried a Test in case something had changed, and that worked too.
Do you by chance have a different computer somehow accessing this account?
Another possibility is that somebody entered the wrong password for your username manually, triggering the lockout. I think it is 3 consecutive fails before Fidelity locks you.
Cal Learner
Cal - Thanks. Yes this was my error. I had two accounts with Fidelity and using SETUP I had updated only one of the accounts. After i updated both, it is working. What is interesting is that all it takes is for one incorrect attempt before the account is locked. THANKS AGAIN for the response and assist!!
DeleteSam A: With the newer versions of PocketSense, PocketSense will not try multiple retries with the same username and password if you set self.skipFailedLogon true in site_cfg.py.
DeleteYou can tell if your version has this feature by seeing if it contains "skipFailedLogon" in site_cfg.py, or just look at the date of the site_cfg.py file. Mine is dated 12/18/2019, but it was not the first to have the feature.
Then your sites.dat must contain this line to implement the feature:
skipFailedLogon: Yes
That line is included in the new sites.template which is what you should base your sites.dat file on. When putting in a new version of PocketSense code, I merge the new stuff from the new sites.template with my old sites.dat. I suspect you may lack that line, and PocketSense tried logging in with your username and password for each of your accounts.
Cal Learner
Cal - Thanks i will check the version and make the changes. Much obliged!
DeleteRe: Schwab and OFX
ReplyDeleteI just moved my Roth IRA, wife's Roth IRA, and a brokerage account from Schwab to TDAmeritrade (and yes, I understand the irony that they are both owned by Schwab). TDAmeritrade OFX transactions are working smoothly. It took about a week for the full transfer to occur (including liquidation of partial shares and transfer of residuals).
In Money, I just renamed the account, imported and voided the new transactions, and manually put in the sale of the small fractional shares. All of my history and everything is still up to date.
Gotta vote with your feet (well, sorta).
I've got a TDAmeritrade account that still works, but anticipate it will eventually be "migrated" to schwab.com. Probably just move my Schwab accounts to Fidelity if they continue on with their "customer centered" improvements.
DeleteI am trying to setup Checking and Savings Account with Bank of America and not having much success. The setting i am using is below. When i run setup, right after i enter user name and password i get an error saying "An error occurred requesting accounts from the site. Please check username and password. I have successfully logged in with the same username and password to ensure i have not been locked out.
ReplyDeleteAny suggestions?
SiteName : BOA
AcctType : BASTMT
fiorg : HAN
fid : 5959
url : https://eftx.bankofamerica.com/eftxweb/access.ofx
bankid : 052001633
brokerid :
ofxVer : 103
appid :
appver : 2400
I don't think BoA allows new users into the ofx servers, only those that were grandfathered in years ago. You can get qfx downloads from the webste though.
ReplyDeleteHaving issues with Fidelity.... Has worked well for years, but this weekend GetData showed an SSL error when attempting to download... I called Fidelity web services, but their only suggestion was to change OFX.Fidelity to OLTX.Fidelity.... That didn't solve the SSL problem.... Anyone else having this issue ???
ReplyDeleteErnie K: I downloaded from Fidelity, using a scheduled task, around sunrise Saturday, and my scheduler does not attempt to download again until early Tuesday, where I pick up any Monday transactions and closing prices. My Saturday and Tuesday downloads worked fine.
ReplyDeleteI suspect that you may have been attempting to download during a maintainance window. I think those tend to be the small hours of Sunday, after midnight just following Saturday, ET.
My sites.dat Fidelity descriptor starts this way:
<site>
SiteName : FIDELITY
AcctType : INVSTMT
fiorg : fidelity.com
url : https://ofx.fidelity.com:443/ftgw/OFX/clients/download
fid : 7776
bankid :
brokerid : fidelity.com
Cal Learner
Having same problem since 11/12. Doesn't appear to be a window log in timeframe. It's all the time and a SSL issue. Validation of OFX files have been issue for several days based on this fidelity log--
ReplyDeletehttp://www.ofxhome.com/index.php/institution/history/449.
MC
What version of Python are you running? Older versions have outdated SSL libraries.
Delete2.7
Delete2.7.x -- The value of .x matters for ssl. It changed around 2.7.13, iirc.
Deleteunknown: I am successfully running Python 2.7.12, and getting OFX from Fidelity. It may be from python.org instead of ActiveState. To get the version number, start Python with the "--version" or "-V" switch.
ReplyDeleteIn Edge, try
https://ofx.fidelity.com/
The purpose is to check that the security stuff is working for you.
Expect a lock icon in front of the URL, and a "Sorry. The page you requested cannot be found on our site. " message.
Cal Learner
Looks like i have 2.7.8. I will try and upgrade Python. Thanks Cal!
ReplyDeleteUpgraded to 2.7.12 and success. Many thanks!!
ReplyDeleteI registered with the Schwab developer site back in october, and tried to input my (static) IPV4 IP's for OFX access. The approval is still 'pending' months later.
ReplyDeleteAnyone have any luck with APPID or other such changes?
Re: Schwab
ReplyDelete> input my (static) IPV4 IP
This just show how utterly insane Schwab's is doing in the name of improving "security". Shutting down legitimate access from their paying customers then on the side allowing backdoor by IP restriction. Once that cat is out of the bag, I am most certain that there will be available unprotected Proxy servers (in the allowed IP list) that one can connect to get access to OFX again.
I totally agree, but that is what was in the profile page for my developer account so I went for it. Since they never approved my request I suppose that is not an option anyways. I would guess it's reserved for big players like Intuit. I'm fine to jump through more hoops but they have given my zero hoops.
DeleteNote on Fidelity, in case everyone runs into this issue again. Two days out of nowhere I started getting the error:
ReplyDeleteYou don't have permission to access "http://ofx.fidelity.com/ftgw/OFX/clients/download" on this server.
No changes on my end. Same scripts, same Python, same password.
Looking back at the PocketSense comments from June, I found the following fix. Add ":443" to the Fidelity URL. My Fidelity URL now reads:
url : https://ofx.fidelity.com:443/ftgw/OFX/clients/download
The fact that others ran into this issue back in June, and I just ran into it now leads me to think they may be doing a rolling wave of updates/changes across accounts.
Carlos,
DeleteThanks for reposting that. I ran into the same problem yesterday -- and I had also just installed on a new computer (to confuse things). I had not run across that URL change looking back. But I made the change and it corrected the problem!
Updates from Fidelity stopped working for me on Dec 5 also. I called Fidelity. They had no clue.
DeleteI had to apply your https://...:443 fix AND upgrade to the newest pocketsense.
We really need someone who is willing to actively manage downloading OFX and provide support. It would be worth it to me to pay $10 a year or something...
I see Microsoft EXCEL 365 has a premium template that provides financial institution updates. Has anyone tried that? Not that I want to pay an active subscription to 365...
Re: Schwab
ReplyDeleteunprotected Proxy server (in the allowed IP list) - bskc.vasvavgrxvaq.pbz (rot13)
https://requests.readthedocs.io/en/master/user/advanced/#proxies
������
DeleteHey Robert,
ReplyDeleteTotally unfair question, as your software is not involved but I don't have anywhere else to turn :( Last month, Money stopped importing transactions for most (though not all) of my accounts. The Money import does not report any issues. I've looked at the OFX files from my bank and I see no discernable format or control value changes. From RBC, my credit card transactions come in, but not my chequing account, however from BMO my credit card transactions don't come in (Only account there). I don't know if it's an issue of the first account failing? I've tried manipulating the order in the .ofx, but so far Money's deemed any of my modified files corrupt. Any ideas?
In sites.dat, there is a variable named combineOFX which defaults to Yes. Try changing it to No. This should let each account load individually to find the problem one.
DeleteThanks Andrew, but I'm not actually using PocketSense for these accounts. I download the .ofx and import directly to Money. As I said, it's not really a fair question, I just don't know where else to turn for someone with Money smarts. Anyway, I've combed through previous working .ofx files with the now failing ones and I see no material difference, and obviously Money hasn't changed, so I'm stumped. Might just have to put a bullet in it and see what else might be out there... :(
DeleteRon Smith: To have somebody take a look at your OFX or QFX file, I suggest
Delete1. Make a sanitized version of the file. Use Notepad or other editor that will not add formatting characters.
1a. Replace your account number content with something like 12345678.
1b. Replace occurance of your name in transactions, if any, with ABCDEFG or some such.
1c. Do not make changes in any characters that look like punctuation or math symbols. Avoid changing amounts, because some of the amounts have a required relationship, such as field 1 plus field 2 always equals field 3.
2. Put the sanitized file into a zip file.
3. Put that zip file somewhere accessible. I suggest tinyupload.com The process will give you a URL for public access to that file. Copy that URL.
4. Paste that URL into a post. Somebody will probably tell you what keeps your OFX file from being accepted by Money.
Note that https://social.microsoft.com/Forums/en-US/home?forum=money still exists for read-only access.
https://groups.google.com/g/microsoft-money is a non-PocketSense Microsoft Money group.
Cal Learner.
Hi Ron,
DeleteYou can also use Microsoft's OFX Analyzer to see any errors in the file. I find it useful to see why a file doesn't work (or why some transactions don't show up--that's usually due to the price*quantity not being equal to the total).
https://microsoftmoneyoffline.wordpress.com/ofx-file-analzer/
Andrew, thanks for the tip on OFX Analyzer. I checked it out. It has no complaints about the files in question. Here's one example:
Delete***OFX 2.0 Add ~2
Running Data Through Parser
No Parse Errors
Reading Data Into Buffer
Tokenizing Data
Initializing Data Structures
Analyzing Data
Verifying Security Types
Verifying Security Uniqueness
Verifying FITID Uniqueness
Verifying Sign Correctness
Verifying Total Calculations
Verifying Other
Done Analyzing File
Cal, just want to confirm before I anonymize and upload: to your comment "what keeps your OFX file from being accepted by Money." - Money does not complain about the files. I get "Import complete" every time. Money even updates the "Bank balance" correctly - I just get no transactions. Let me know if you think the exercise is still worthwhile
Just as a note: I'm using Money 2006. IIRC, there were some constraints with the sunset version w.r.t. Canadian users, so I left that alone
DeleteHi Ron,
DeleteSince Money doesn't seem to be complaining about the imported file(s) could it be that you have inadvertently applied one of the filters?
e.g. Unaccepted Transactions or Unreconciled Transactions
Make sure that the All Dates and All Transactions options are selected from the 'View' drop-down list for the registers in question.
-Kevin N.
Ron Smith: Still worthwhile.
DeleteRegarding Money Plus Sunset, you might installing that on a different computer to check to see if there are issues. Money Plus fixed Spinoff, Backup to USB flash drive fixed with >2GB free, it honors number of backups setting for scheduled backup, and probably more.
I don't know if there would be issues with Canadian files. I am thinking Money 2006 for Canada may have only used "Essential" reports and other thing, and not "Advanced". I am not sure.
Cal Learner
Just to close this thread off, I also posted in the Google group. As I result I tried Money's "repair money file" feature, which seemed like it may have done the trick, but not completely, i turned out :( . As of now, I've started a fresh .mny file. So far, so good.
DeleteRon, because you aren't using Pocketsense, you aren't getting the benefits of "scrubbing" the ofx files. So look for those items, for instance with Schwab...
ReplyDeletePocketsense scrubber for schwab:
# 17-Feb-2013*rlc
# - Added scrub routine for CORRECTACTION and CORRECTFITID tags (not supported by Money)see if those are in the problem files.
- ameridan
Thanks Ameridan. It's been some 14 years, so recall is a bit sketchy, but I don't think (?) the Canadian banks expose the endpoints pocketSense would need. I'm game to try, if anyone has a sample for RBC or BMO. In any event, Money does not complain about the files. When I monkey around with them to test out some theories, Money is pretty consistent in complaining the (modified) file is corrupt, but using the files as provided by the bank , Money reports "import completed" every time.
DeleteAmex card transactions download stopped working since yesterday. https://online.americanexpress.com/myca/ofxdl/desktop/desktopDownload.do?request_type=nl_ofxdownload responds with a timeout or "Internal server error".
ReplyDeleteThis was happening to me, too, since yesterday. But it seems to work fine now.
DeleteConfirmed, Amex is back on. Seems like a temporary glitch on Amex end.
DeleteFidelity Investments not working this morning, Jan 28. Worked yesterday.
ReplyDelete<CODE>15500
<SEVERITY>ERROR
<MESSAGE>Error occurred logging in
I hope it is transient and not due to intentional changes.
Cal Learner
Fidelity Netbenefits (their 401(k)/403(b) servicing broker) still works.
DeleteI can't connect to USAA today (I was successful yesterday).
-Andrew
Fidility is and was working fine.
DeleteIt was totally my error. I had changed my username. I messed up in my method of making the change in my unencrypted ofx_config.cfg causing my changes to not be made.
Cal Learner.
I noticed that USAA just updated their OFX servlet location as of 1/25/2021. I can't find the latest URL anywhere. Would someone be kind enough to post the URL please?
ReplyDeleteThank you!
-Tony
MoneyDance shows the url as https://service2.usaa.com/ofx/OFXServlet
DeleteI don't use USAA, so I am unable to tell if the is the old or new one.
You can always see what they use at http://moneydance.com/synch/moneydance/fi2004.dict
MoneyDance users are discussing this at https://infinitekind.tenderapp.com/discussions/online-banking/17962-cant-download-usaa-transactions
DeletePost #12 says
USAA has a new direct connect procedure-- (USAA Federal Savings Bank-New) is the new institution to setup and it requires a UserID and Pin assigned by them-not your old USAA member number. This was changed as of yesterday.
Post #17 has a link to quicken's solution
Yeah, unfortunately, for Pocketsense users, none of the information will help. Since I use Microsoft Money, I have to manually configure PocketSense for the updated ofx server. It seems that some folks over at MoneyDance are still having issues after following the suggestions from the tech support team. Hopefully, someone posts the updated bank information.
DeleteIn the meantime, I'll check out that MoneyDance page.
-Tony
So, it looks like USAA is going to a new system where the application needs to first authenticate with their servers to request application access as shown in this page: https://infinitekind.tenderapp.com/discussions/online-banking/17962-cant-download-usaa-transactions
ReplyDeleteApparently even MoneyDance users are having problems with the site now. I'm wondering if PocketSense will be able to work with a site that requires this new type of authentication method.
One more update...I discovered that USAA moved the ofx server to the following URL: https://df3cx-services.1fsapi.com/casm/usaa/access.ofx
DeleteThis is according to this page: https://lists.gnucash.org/pipermail/gnucash-devel/2021-January/045665.html
Not sure if that's enough for PocketSense to be able to reconnect to USAA but I would be more than happy to throw in some coffee change if you can get it fixed!
More information is coming to light. It would appear that USAA has switched to OAuth using Intuit's servers in the transaction process. The developers of Gnucash and Moneydance are both trying to reverse engineer what is going on behind the scenes. If this is indeed the case, will Pocketsense work? The software would need to first connect to the Intuit servers and then request a password and pin through the software. USAA would then add the software to the security profile on the account.
DeleteJust checked OFX Home status for USAA FSB (https://www.ofxhome.com/index.php/institution/view/483). Though that web page suggests the published OFX link information is "Validated", a quick peek into "View History" shows the following:
Delete...
SSL passed validation on January 26th, 2021
OFX passed validation on January 26th, 2021
SSL passed validation on January 27th, 2021
OFX failed validation on January 27th, 2021
...
The OFX status is "failed" ever since January 27th. I will join the throngs calling USAA next week to request some support.
USAA OFX has failed validation on OFC Home every day since 27 Jan 2021.
ReplyDeleteWhat does NOT work:
ReplyDeleteGetting new Access ID/Password (from https://api.usaa.com/auth/oauth/authorize?client_id=9fef871a-f181-4734-b603-9385c8db23b3&redirect_uri=https://df3cx-services.1fsapi.com/casm/usaa/connect&scope=usaa.profile.member.read+usaa.bank.aggregation.read&response_type=code&code_challenge=my_challenge&state=my_state&code_challenge_method=plain
Setting up a new site in sites.dat with new server location and 103 appid
```
SiteName : USAA Banking NEW
AcctType : BASTMT #bank
fiorg : USAA
fid : 24591
url : https://df3cx-services.1fsapi.com/casm/usaa/access.ofx
bankid : 314074269
brokerid :
ofxVer : 103
appid : QWIN
appver : 2700
mininterval:
```
Clearly something else going on.
I'm hoping that Robert can chime in and see if he can take a look and see what's going on. It would appear that sending angry messages to USAA isn't cutting it. By the way Ron, I see a client_id in the URL above. Was that copied from Quicken? I would imagine we would need to generate a unique id if we try to request an app PIN and password.
ReplyDeleteThat URL was copied from another site. And you may be correct, but you need to navigate to the URL only after you've logged into USAA. And after doing that, Quicken showed up as a connected app on the Security page of my Profile in USAA.
DeleteAhh okay, I see what you mean. I was able to generate the ID and password, but like you said, you can't get PocketSense to work.
ReplyDeleteI have noticed something else that may be related to that URL ID. The documentation states that if you lose your id/pw, you can get another. However, if I disable the "connected app" in USAA, and then use that link to get "another", the returned ID/PW is identical. So that client_id may be Quicken generated and have something to do with that. I also noticed, this morning, that the Quicken as a connected app in my profile has disappeared! With all the back and forth testing, I'm not sure if that was something I did or something they did. But since the URL always seems to generate the same ID/PW combination, I was going to disconnect it anyway, on the off chance that someone would get the same combo as me using that link.
Deleteon, and Capt_Ahab: I am interested your observations, mainly in that this could propogate to other FIs. I fear this ID is somehow signed by Quicken, and even if you could figure out the protocol, that ID would stop working when your Quicken subscription expired. I may be overly suspicious.
DeleteCan you create another Quicken file for a different identity, such as a spouse? I suspect that will generate a different ID despite using the same URL.
Cal Learner
I myself did the same thing after testing to see if I could at least "trick" USAA into thinking that I was registering the Quicken app in hopes of using the resulting id and pin for OFX authentication but unfortunately it's not working. Robert's latest post below on Feb 2nd seems to indicate that this issue is beyond the scope of PocketSense. I am keeping an eye on the Moneydance tech support forum since that product also relies on the OFX standard to connect to financial institutions.
DeleteI've also been monitoring the USAA Community forum and there is mention of this FDX standard that is a coalition of companies to include Intuit and USAA. I don't know if this is a replacement for OFX or what but here is a link to the page showing USAA as a member: https://financialdataexchange.org/FDX/The%20Consortium/FDX/The-Consortium/Members.aspx?hkey=362ecd23-b752-48aa-b104-a99e916276c8
I don't have much to add re the USAA issue, except that the OFX request/response schema used by the scripts doesn't support OAUTH. The scripts are based on the OFX 1.x Standard, where authentication is performed via user/pw, plus a unique client_uid provided by the bank on first connection when ofxVer=103. Technically, there would be little difference for subsequent requests if the bank instead provided the access token using a separate / out-of-band connection. I suspect that something like this may be happening w/ Quicken, where the token is being assigned independent of ofx requests.
ReplyDeleteRobert,
ReplyDeleteSince last week USAA Bank added account and pin numbers to down load statements.Please help.
jmf
Robert,
ReplyDeleteSince last week USAA Bank added account and pin numbers to down load statements.Please help.
According to a commenter today on the MoneyDance blog: "According to support, their new authentication tokens have to be built into the software". I wonder if that is something that could be added to the PocketSense scripts. Of course, I have no idea exactly what USAA means by that nor how it could be accomplished. Maybe more information will become available in time.
ReplyDeleteRe: USAA
ReplyDeleteI don't have USAA but a Moneydance user sent me an email and mentioned that manually CSV download is available (as current work-around) and if I can help him convert the CSV into OFX. I have some codes that I was able to re-use and create a command-line tool for him. If this is something that you can use, see http://bit.ly/3cPajDZ
I already have a CSV=>OFX converter, but the entire process -- Log in to USAA Web interface; select account: select the date range to be downloaded; download; convert; open with MSMoney; check the transaction matching for four accounts (two of which are pretty active) takes a heck of a lot longer than merely opening PS and running the script. Not a good long-term solution for me as it takes much too much time and personal intervention. I have applied for a Fidelity credit card (where I already have brokerage and retirement accounts). Once that is approved, if USAA remains recalcitrant, I will be moving my banking business.
ReplyDeleteRon said "I already have a CSV=>OFX converter, but the entire process -- Log in to USAA Web interface; select account: select the date range to be downloaded; download; convert; open with MSMoney; check the transaction matching for four accounts (two of which are pretty active) takes a heck of a lot longer than merely opening PS and running the script."
ReplyDeleteRon, a csv or qif to OFX converter must produce a transation id. Besides being unique for the transaction, the same FITID should be produced if the same transaction is re-downloaded. One way to do that is to include date, amount, and payee in the generation. In addition, the download should be checked for two identical transactions, and generate two different FITIDs if that happens.
The reason I bring that up is that if you are spending time matching transactions, that may be because your converter is not doing a good job generating FITIDs.
My current bank, Citibank, lets me download OFX by web, but it uses a different FITID each time the same transaction is downloaded. I don't have a lot of transactions at that bank. I have a Citibank credit card with them, and their OFX works flawlessy with PocketSense.
I have the Fidelity credit card too, and it works well. PocketSense cannot download the transactions, but does download the payments, which I convert to transfers to the CC. The CC transactions can be downloaded via web. The FITIDs are good. The payments match the already-there transfers in Money.
I hope that USAA is willing and able to fix this. I agree with those that say that the change was probaby motivated by Quicken. I suspect it was Quicken that provides the OFX/QFX server software, and maybe more. So the new IDs would serve as keys. I also suspect the IDs would stop working when your Quicken subscripion expired.
Cal Learner
It seems like someone finally cracked the code to at least get gnucash to interface correctly with USAA. Can someone get PocketSense to adapt to the new method? Here is the link to what they discovered: https://lists.gnucash.org/pipermail/gnucash-devel/2021-February/045690.html
ReplyDeleteThis comment has been removed by the author.
ReplyDeleteHey Robert, are you the one who maintains the PocketSense script or is it someone else? It looks like we have the fix but the script would need to be programmed to change the way that it sends the request to the OFX server. I'm guessing this would be accomplished in the ofx.py module.
ReplyDeleteTo whom it may concern (Python guys):
ReplyDeleteCall USAA Customer Service (210-456-8040) and ask for the "IT Department." IT is very knowledgeable as to what is required to comply with new changes (made without notice), and familiar with "Microsoft Money." Thanks
USAA Bank handling development idea:
DeleteI, or somebody else who can generate Python code, but is not a USAA bank user, could help get this USAA thing going. I am pleased to read that the USAA tech people are supportive of this. I was afraid that would not be the case.
1. How would I get a request OFX into a file as PocketSense will send it to the FI? I know that I should be able to figure this out, and I probably could eventually, but somebody probably already knows.
2. What is the current site descriptor for USAA bank?
3. Given those, I would propose to post a USA request file to tinyupload.com, with pretend account number and password. Then I would propose that somebody modify that with Notepad etc to
what they think would be the request file that meets the new criteria.
4. I would modify one or more of the *.py files to make the modifications. The modifications would not be as elegant as the rest of the code; it would be less object-oriented and more Kernigan and Richie style. It might involve a request scrubber, or be done some other way.
5. Some user would inspect the modifications to see there was nothing nafarious in my modifcation, and try it out.
6. If that code works, at some point, Robert would probably incorporate the features in a more readable style. That could involve adding one or more possible entries to the sites.dat descriptors.
Cal Learner
I did some experimenting. I put 3 files into a zip file:
Deleteofx.py (modified to write the bank request to a file for debugging)
reqq (the request file I got by logging)
usaanote.txt (notes)
SHA-1:
533c7d5d80e1d0cac51d0e1cb600b700d5dbd789 usaa.zip
If SHA-1 doesn't interest you, then ignore it.
File was uploaded successfuly on TinyUpload.com server.
You can download it from address: http://s000.tinyupload.com/?file_id=45164446686448891753
Cal Learner.
There is a program ofxTools (https://github.com/csingley/ofxtools) that has recently been updated to use the new USAA protocol. There is also a thread regarding Gnu-Cash (https://lists.gnucash.org/pipermail/gnucash-devel/2021-February/045704.html] which discusses the new `site.dat` entries for the new USAA protocol. These might be useful to someone knowledgeable. Unfortunately, ofxTools requires Python3.6 or 3.7+, and PocketSense won't run under Python3+.
ReplyDeleteYeah, the other issue with ofxtools is that it doesn't appear to be as user friendly as PocketSense. It doesn't have a menu driven setup and you basically have to give it parameters. I suppose you could create a batch file to run it with the needed parameters but I'm not sure how easy it would be overall. Ideally, I'm hoping that Rob would be able to look at the notes from the gnucash forum and see if he can implement those changes in his script. Hopefully this won't require too much time to implement.
ReplyDeleteJust a quick update....One of the CSRs at USAA has posted a link that will let you generate your own clientID, Access ID, and Access PIN. You have to log into USAA first and then open this link: https://www.usaa.com/accessid After accepting, you will see the clientID in the URL on the page with the ID and PIN. I would copy that and save that with the other important information.
ReplyDeleteHi Rob, I was wondering if you have plans to modify PocketSense to work again with USAA. At this point, I think we have figured out how to grab the necessary information (clientID, AcceessID, and PIN) so that removes a huge issue. The PocketSense software would just need to be able to send the request to the new USAA server in the format as specified in the gnucash forums (as Ron has linked). You're basically changing the format a bit and sending the clientid value in the script. Please let me know if this is workable or not. Thanks!
ReplyDeleteI don't use USAA so no way to test, and am unable to focus on it at the moment. However... if someone can write out explicit rules required for USAA ofx connections then I'll see what's needed to fold into the scripts. By "rules", I mean exact differences vs standard OFX requests. Assuming it's still OFX 1.x, then I suspect something very simple like "use this code for username, this one for pw, and this one for clientUID". Of particular importance is whether clientUID is server-assigned or must be entered by the user. The scripts currently only support auto-assigning the clientUID on first connect w/ the server. Keep in mind, though, that changes that are too specific may break other connections, but we'll see. - Robert
ReplyDeleteHere are the instructions for the additional steps:
ReplyDeleteSTEP 1
Enter your logon information.
Go back to Quicken Direct Connect and enter your Access ID and Access PIN (obtained from https://www.usaa.com/accessid).
STEP 2
Select "Connect."
Follow the on-screen prompts to set up and download your accounts.
Hi jmf, I am familiar with this process but have you had success getting PocketSense to download transactions?
DeleteRob,
I'll try to copy and paste some of the pertinent information from the gnucash forums where they go into detail on the format of the ofx request. This also includes updating the USAA ofx server URL.
Here is some good technical info from the gnucash forum discussion that goes into the ofx request format:
ReplyDeleteI got this working in my software with some help for the info on this list.
Here is a write-up:
USAA's changes to their OFX interface
-------------------------------------
On 2020-01-26, USAA's previous OFX interface (
https://service2.usaa.com/ofx/OFXServlet) stopped working. It seems like
they switched to a new interface through a tech provider to replace their
previous login method (with your website credentials) to an app-specific ID
and password. This is a good move for security, but it was done without
notice, it seems, to anyone but Quicken.
>From some internet searches, I found some people on the right track to
fixing this on the GNU Cash development mailing list:
https://lists.gnucash.org/pipermail/gnucash-devel/2021-January/045664.html
They were able to determine that USAA was:
- using a new OFX endpoint:
https://df3cx-services.1fsapi.com/casm/usaa/access.ofx
- using a new OFX Org ID: USAA Federal Savings Bank
- using a new OFX FID: 67811
Additionally, someone on the USAA forums was about to extract and post the
link to generate an App ID and PIN:
source:
https://communities.usaa.com/t5/Finances/USAA-Creates-Quicken-Monopoly/td-p/243850/highlight/false
Authorization link: https://df3cx-services.1fsapi.com/casm/usaa/enroll
However, with a lot of trial and error I still wasn't able to hit this new
endpoint successfully. So I decided to give the devil his due and
temporarily got a Quicken subscription and setup an SSL man-in-the-middle.
The new OFX interface is *very* finicky, so you basically have to input
everything exactly the way it expects it. Here is an example of an account
listing query that works:
echo -en
"OFXHEADER:100\r\nDATA:OFXSGML\r\nVERSION:103\r\nSECURITY:NONE\r\nENCODING:USASCII\r\nCHARSET:NONE\r\nCOMPRESSION:NONE\r\nOLDFILEUID:NONE\r\nNEWFILEUID:NONE\r\n\r\n\r\n\r\n\r\n20210207031942\r\nXXXXX\r\nNNNNN\r\nENG\r\n\r\nUSAA
Federal Savings
Bank\r\n67811\r\n\r\nQMOFX\r\n2300\r\n1955A543-B071-455E-A31E-73CC7C493D68\r\n\r\n\r\n\r\n\r\ne39add7d-2085-4504-b9ee-be37927de39c\r\n\r\n19900101\r\n\r\n\r\n\r\n\r\n"
| curl -isS -X POST -H "Content-Type: application/x-ofx" -A InetClntApp/3.0
--data-binary @- https://df3cx-services.1fsapi.com/casm/usaa/access.ofx
Note you have to change the XXXXX and NNNNN to the App ID and PIN you get
from the link above.
Some things I've found through trial and error:
- The OFX elements must be separated with "\r\n". This is dumb, but true.
No spaces. No simple "\n". Exactly "\r\n".
- The APPID "QMOFX" and APPVER "QMOFX" work. Others I tried did not.
- The CLIENTUID "1955A543-B071-455E-A31E-73CC7C493D68" works for me. It
must be uppercase. This might be particular to your account. If so, you can
find it looking at the OFX logs from Quicken.
- TRNUID must be present, but an UUID will do.
- DTACCTUP: The value "19900101" works. The value "19700101" does not. The
value "19900101000000" does not.
- You need the "Content-Type: application/x-ofx" header
- You need the User-Agent "InetClntApp/3.0". This is what Quicken for Mac
sends.
It also seems their gateway will under some conditions put your IP on a ban
list. If you are testing, you may want to spin up an AWS instance or
something.
Thanks to Bob White on the GNU Cash list and RDD! on the USAA Forums for
the breadcrumbs. No thanks to USAA for swapping out their functional
interface with absolutely no notice or documentation and pretending like
Quicken users are the only customers of any importance. Please just don't
break our software again... at least for awhile.
- Scott McRae
The only change is that it appears that the clientID can be found within the URL after going to https://www.usaa.com/accessid and going to the page that issues the AccessID and PIN. This could be an additional value that you can add to the sites.dat file that will need to be present if the user is trying to get transactions through USAA.
DeleteOne more thing, if you need someone to test out the PocketSense script with USAA, please let me know and I can run it and let you know what happens.
DeleteScott McRae:
DeleteIf you want to show OFX/ SGML in a post, you can change all < characters in your post to <
Similarly change all > characters in your post to these 4 characters: >
Did you look at my February 10, 2021 at 4:49 PM post?
Cal Learner
The clientID (UUID) can be edited in the `connect.key` file using a text editor like notepad++.
ReplyDeleteI skimmed over the link above (https://lists.gnucash.org/pipermail/gnucash-devel/2021-February/045704.html) and summarized the following. Please review / commment.
ReplyDelete1. They appear to have a tool where you login and generate an appID and token (PIN), which would be used as the userID and userPW.
2. OFX elements must be separated with "\r\n".
4. DTACCTUP must be yyyymmdd format.
Site info:
url : https://df3cx-services.1fsapi.com/casm/usaa/access.ofx
fiorg : USAA Federal Savings Bank
fid : 67811
bankid : 314074269
appid : QMOFX
appver : QMOFX
*ClientUID*: It's not clear whether clientUID is assigned during connection or pre-assigned by the bank? If someone can use their online tool to see what it gives you, it answers the question. If they only give the appID and PIN then the clientUID is created during initial connection, per OFX 103.
Hi Robert,
DeleteThe ClientUID is contained in the URL when you create your access ID/PIN. After enrollment, it directs you to a webpage with your ID/PIN. The URL had the following form for me:
https://df3cx-services.1fsapi.com/casm/usaa/connect?code=********-****-****-****-************&state=my_state&scope=usaa.profile.member.read+usaa.bank.aggregation.read
where * contained my hexadecimal clientid.
This comment has been removed by the author.
DeleteI have managed to get a successful connection using edited pocketsense scripts and clientuid, access id and access password from USAA. I'm not getting account info returned because I believe I've tried and failed too many times via trial and error from same IP. I'm going to try what I have from another IP address.
ReplyDeleteHi John, when running the PocketSense script, are you putting the clientuid into the connect.key file? For me, I have two entries in there so I'm unsure which one to edit if this where you are making the change.
ReplyDeleteUse setup.py to find the clientUID (connectKey). The file is partially encrypted on purpose, but not the key itself. Use 2. List Accounts, and Y to "show connection keys". You can then change that one in connect.key to whatever you want. The connectKey is created once for an account connection, and would only be regenerated if the account is deleted in setup and then recreated.
Deletep.s. Note that the key is between single ' quotes. Don't delete those. Also, I'd make a copy of the file before editing, and only use something like Notepad++ that doesn't add hidden CR/LF chars.
DeleteOne other question...Are you using all caps for the clientID? Also, I tried to set the username and password again through PocketSense and it's not taking the new AccessID and PIN. Is that what you're using for username and password?
ReplyDeleteI added an additional field to the USAA site in the sites.dat file which contains my clientUID. This value was obtained from the URL where you get your new access id and pin. This is in the URL prior to clicking the Allow button. What Andrew is pointing to above is not the client did. When this gets added to the connection string it must be in all caps. My code mod converts the data in the sites.dat field to uppercase for the clientUID field. I send this as the clientUID rather than the one the scripts calculate.
ReplyDeleteOne more question John, where did you get the edited version of PocketSense that you are using? I think that's my issue because when I try to change the username and password to my AccessID and PIN, the script informs me that there is an issue with the username and password.
ReplyDeleteAhh...would you mind sharing your code mod then? I am using the standard PocketSense script at the moment. Also, could you provide an example how you insert the clientUID field into the sites.dat file? I simply overwrote the information between quotes for the exiting one generated by PocketSense. Also, what are you using for username and password. Is this this now the AccessID and PIN or just the normal USAA username and password we were using before the change?
ReplyDeleteI'm modifying the Pocketsense code on my home computer myself. The current released code isn't going to support the changes to the USAA site. I'm a software analyst/developer with background in c and c++. I'm picking up Python from looking at the scripts.
ReplyDeleteYou have to use the new user id and password you get from the USAA site. The clientUID needs to be added to the site definition for USAA only.
ReplyDeleteAhh, would you mind sharing what changes you made to the script? I have written some simple python scripts but nothing like PocketSense.
ReplyDeleteAhab - Hang on a minute and I'll tell you what changes I've made.
ReplyDeleteOkay cool. Thanks!
DeleteAhab - Sorry, I got tied up at work and didn't get a chance to post my changes. Here is a brief synopsis of what needs to be done to set up a connection to the new USAA OFX server.
ReplyDelete1) Log into www.usaa.com using your normal username and password combo
2) Navigate to www.usaa.com/accessid to enable Quicken access and get your new access ID and PIN
3) The URL of this page contains your unique client_id, which you will need to include in the sites.dat file
4) Once you provide access permissions and click the "Allow" button you will be taken to another page that identifies your access ID and PIN. These two values need to be supplied as the username and password for access to USAA through the Pocketsense scripts. You will be queried for these when you attempt to set up an account the first time for the new USAA.
5) Create a new site definition for USAA using the info supplied in one of the messages above. Add a clientId field to this site definition that contains the client id that you obtained in step 3.
6) The ofx.py, Setup.py, and site_cfg.py files have to be modified in order to get them to format the OFX string to be sent to USAA correctly. The site_cfg.py file needs to be updated to read a new clientId field from the USAA site definition. The Setup.py file needs to be updated to create a different DTACCTUP value for just the new USAA account. This value should be 19900101. The value used by other accounts is 19700101000000. The ofx.py file needs to be updated to use the clientId for USAA rather than the clientId from the connect.key file. This file also needs to be updated to insert 'User-Agent', 'InetClntApp/3.0' rather than 'User-Agent', 'PocketSense' in the OFX header. I suspect there might need to be some changes to GetData.py but I'm stuck at this point because I can't get account info returned due to having tried and failed too many times to get the correct string sent over.
7) Get account info downloaded from USAA and into the Pocketsense database.
8) Try to retrieve transaction data from USAA using the updated scripts (might require some additional updates) and downloaded accounts.
".. I can't get account info returned due to having tried and failed too many times to get the correct string sent over."
DeleteSuggestion: check your profile on USAA, revoke current Quicken app access, then get another ID and PIN(?).
My IP is whitelisted (i.e. blocked) by the third-party site that's using the Imperva Incapsula cloud management software. I verified this today with the Imperva operations center. I have to somehow get USAA to remove my IP address from the blocked list (easier said than done given that most CSRs at USAA don't know what whitelisting is or even what the Incapsula software is).
DeleteAfter looking at the details of what needs to be done, I think I will just need to work with Rob and hope that he can decipher what changes seem to have worked for the folks at GNUcash. If you need me to try out a script change for USAA Rob, let me know and I can run and let you know what happens.
ReplyDeleteThanks!
FYI, another checklist for things that I had to adjust in getting my self-build tool hleofxquotes to be able to download USAA OFX files.
ReplyDeletehttps://bitbucket.org/hleofxquotesteam/hleofxquotes/wiki/USAA
Look for section "Background notes" toward the end.
To all of the above: my humble thanks for your efforts to straighten out the mess that the corporate idiots running USAA these days created. I love using PocketSense but need step-by-step instructions for a compleat idiot such as myself on what/which/where/how to edit files in order for it to resume functionality. Why the powers that be at USAA chose to give Quicken a monopoly on their ofx facility is anybody's guess.
ReplyDeleteToday I was able to get an access ID and PIN from https://df3cx-services.1fsapi.com/casm/usaa/enroll, so we know that is no longer an obstacle. I do need somebody knowledgeable to spell out the rest of the fix, i.e., which files to edit and explicitly how, with step by step instruction from someone who will take pity on me. I'm sure Robert will be might grateful, too.
Respond to this and I'll tell you specifically what changes need to be made to the scripts. Besides the access id and PIN, you also need the client id from the USAA site.
DeleteP.S. When you get an access ID and PIN from USAA's link you'll get an acknowledgment email:
ReplyDelete"Quicken has been connected to your USAA account. To manage what apps are connected to your account, view your USAA Profile."
Your profile in USAA shows that you are allowing Quicken access your name, home address, phone #'s, email address, all account numbers, transactions and balances. I'm not too sure about that and may need to take my banking elsewhere, as much of a pain in the neck as it might be to switch.
I believe the info provided thus far is enough to add support in the default scripts w/o being specific to USAA. Note, however, that *all* OFX connections provide access to personal info you've listed. Also, it appears that when you generate the ID and PIN, you must copy the clientUID in the address bar at that instant. That key must match the ID and PIN on future connects. I can't test, but that's my take.
DeleteThank you Robert.
DeleteAlso, for the partially paranoid such as myself I just re-discovered that there are check boxes on the ID and PIN "application" page:
Allow Quicken to access your USAA information
Quicken is requesting permission to access your:
[]Account Owner Information
[]Bank Account Numbers and Details
I left the Account Owner Information box unchecked and will attempt to use the ID and PIN set provided and determine whether connectivity can be achieved without it. I see no need to volunteer my name, home address, phone #'s, etc. to Quicken unless absolutely necessary.
P.S. Please disregard what I just wrote above about leaving a box unchecked; I thought it worked initially but on multiple re-test attempts it errors out and will not provide ID or PIN numbers unless both are checked.
Delete@Robert. I believe I saw someplace that, for USAA, the ClientUID needed to be all caps.
DeleteRobert, the clientUID doesn't change. If you go back to that page, the same clientId appears in the URL every time you visit it.
ReplyDeleteHi Robert, John is right...Once you are able to get the clientUID from the site, from everything I've read, this value is permanent (as well as the AccessID and PIN) for all future connections.
DeleteI've updated the scripts to support a few extra site parameters, driven mostly by the USAA request, and a DEV version is available for testing. Use the site entry below, including your clientUID value. User-name (AccessID) and password (PIN) must be defined in the account entry via Setup.py (add new account). I also included some updates from a long standing "to do" list, which I'll explain when posting a final version. - Robert
ReplyDeleteSiteName: USAA
url: https://df3cx-services.1fsapi.com/casm/usaa/access.ofx
fiorg: USAA Federal Savings Bank
fid: 67811
bankid: 314074269
appid: QMOFX
appver: 2300
ofxVer: 103
dtacctup: 19900101
userAgent: InetClntApp/3.0
clientUID: your_value_here
Hi Robert! Thank you for getting PocketSense to work with USAA. I did run setup.py and selected to create a new USAA account and edited the sites.dat with the specified information above. I tried using an all caps clientUID as well as all lower-case clientUID. I keep seeing the following message now: "An error occurred requesting accounts from the site. Please check username and password." I'm not sure exactly what is causing the issue however. Would you happen to know if there is someplace I could check to see what is going on? I did make sure to use the AccessID for the username and the PIN for the password also.
ReplyDeleteThanks in advance!
Ahh...I browsed to the site and it looks like it's blocking my IP...that could very well be the issue...
DeleteOf course, I'm not privy to what's been done up to this point that may have caused an IP to get blocked. If that's the case, then it would have to be unblocked by someone on their side, unless there's a timeout of some type. Another option to consider, and won't hurt if you're already blocked, is to manually enter the account # rather than depending on them providing a listing during setup. That feature is separate from OFX requests, and may not be supported on their new server. Basically, answer "Y" to continue adding an account and enter manually... and test when prompted.
DeleteYou can't open the ofx endpoint in a browser. That's prohibited by the rules for the site. If your IP was blocked you'd be getting the robots response that I am getting.
ReplyDeleteAhab - Which page are you getting the client id from? It should be the page that has the checkboxes and Allow button, not the one that has the access id and PIN displayed.
ReplyDeleteOkay, I made sure to grab the clientUID from the screen where it has the checkboxes and the allow button. I put this in the sites.dat file and went to set up a new USAA account. This didn't seem to do anything. Can anyone confirm if they managed to get the DEV version of PocketSense working with USAA?
ReplyDeleteI was not able to access account.
ReplyDelete- I downloaded the new DEV and installed it over a copy of my functioning PS script
- I edited sites.DAT as shown by Robert:
SiteName: USAA NEW
url: https://df3cx-services.1fsapi.com/casm/usaa/access.ofx
fiorg: USAA Federal Savings Bank
fid: 67811
bankid: 314074269
appid: QMOFX
appver: 2300
ofxVer: 103
dtacctup: 19900101
userAgent: InetClntApp/3.0
clientUID: my ID with all CAPS
I then attempted to set up automatically:
---------------
Configure account for USAA NEW
User name : xxxxx
Account password: xxxxx
An error occurred requesting accounts from the site. Please check username and password.
Continue configuring account (Yes/No): [N] y
*****************
NOTE: If the account number is masked (e.g., XXXX-XX-1234),
you MUST manually enter the account number
*****************
Enter line #, *or* the actual account number
Account List
------------
Account # : xxxxx
Replacing xxxxx for USAA NEW
Do you want to test transaction downloads for the new account now (y/n)? y
USAA NEW : xxxxx : Getting records since: 20210116
local variable 'query' referenced before assignment
An online error occurred while testing the new account.
-------------------
Hi Ron, I haven't tried proceeding with the manual setup. I'll give that a try and see what happens.
DeleteHere's the result I get when proceeding with manual setup and run the test at the end:
ReplyDeleteTest account #: [0] 1
Test USAA | XXXXXXXXXX (Y/N)? y
USAA : XXXXXXXXXX : Getting records since: 20210116
local variable 'query' referenced before assignment
An online error occurred while testing the new account.
"local variable 'query' referenced before assignment" looks like a bug. Stay tuned...
ReplyDeleteWill do! Thanks Rob!
DeleteI suspect the issue is that the site parameters I listed above are only the fields I knew. I omitted account type, since I didn't know what it would be. For example, for a bank account:
DeleteAcctType: BASTMT
The AcctType field is required.
Hi Robert,
DeleteI did add AcctType to BASTMT and that error message went away. I then received some kind of OFX error message that states the following: Request unsuccessful. Incapsula incident ID: 1246010460002156800-3480998060623239.
Any idea what may be going on?
Thanks again!
One other note...I did have to manually enter the USAA checking account number as it still did not take the username and password for auto-setup. I'm not sure if that's supposed to happen or not.
DeleteOkay quick update....It works! The issue was that you MUST use the ID given on the page WITH the Quicken AccessID and PIN. If you use the clientID on the previous page, it doesn't work. I first tested with that java ofx tool first and then confirmed it's working with PocketSense as well.
ReplyDeleteThank you very much Robert!!
Which java tool are you referring to?
DeleteI think this is the Java tool (I use it for Schwab now): https://bitbucket.org/hleofxquotesteam/hleofxquotes/wiki/USAA
DeleteYeah, that's the one but I really like PocketSense better because you can just run the script to grab the bank info after it's been set up. Since hleofxquote is GUI based, I don't know if there is a way to use it in a startup script for Microsoft Money. I don't know enough about it though to say for sure.
DeleteBy the way, I'm sending in a donation. If any one else wants to thank Robert for his hard work for fixing the USAA issue, please consider sending him some money. (Link here: https://sites.google.com/site/pocketsense/aboutme-1?authuser=0 )
ReplyDeleteThanks!
Ahab: Tried setting up and I'm getting the "robot" response as an invalid OFX. (also has the `Request unsuccessful. Incapsula incident ID` line in the returned text)
ReplyDeleteFor clientID I used the `code=` segment of the URL on the same page as myu AccessID and Access PIN.
I did not upperCase it. Did you?
I did add `AcctType : BASTMT #bank` line to the sites.dat entry
According to Fleming, that response means my IP is being blocked. I think there is a time out for that, but I don't know what it is. Any suggestions appreciated.
Yes, I would use all uppercase characters for the clientID as this is what I used. It sounds like you have the correct one if you copied the 'code=' segment all the way up to the '&' in the URL. As far as the IP being blocked, I would just try it again and if it doesn't work, wait an hour or so and try one more time. If everything is working correctly, PocketSense will prompt you to select your account number after putting in the AccessID and PIN.
ReplyDeleteSadly, it is still not working, in that I get the "Robot" response. I even came in from a different IP address (using a VPN) and got the same response. So my problem is not IP blocking. I don't have more time today to work on this, maybe later in the week.
DeleteRon - The Pocketsense scripts attempt to send a given request to the OFX server up to three separate times. I believe the one that is giving the "Request unsuccessful" response is actually from the third attempt. This one is going to fail every time. Success has to occur on one of the first two tries to get a successful response. If you look at the responses from the first two requests I believe you'll just see an empty robots header. This empty header is the indication that your IP is blocked.
DeleteJohn,
DeleteThat may be. But since I started a series from a fresh IP address via a VPN, that does seem to point to a problem in my setup, and not IP blocking as the cause of my problem. Especially since Capt_Ahab, and you, both seem to have things working.
Ron: It sounds like something isn't quite right w/ your settings. I'd double-check every entry, especially the UserID, PIN and clientUID values (I believe clientUID must be UCASE?). In fact, case often matters for OFX params, including the url itself, since it's included in the request header. Verify that you didn't copy bracketing symbols from the url (i.e., leading/training chars). Grab the latest beta below for testing. If it still fails and you want to dig further, enable debug in control2.py by uncommenting the Debug=True line.
DeleteRon - I can't get an account list nor download transactions, so I don't have things working. You might try swapping to debug mode and then posting your ofx query string with clientuid, username, and password redacted/removed. It would probably be best just to x them out, so we can see the string that's being sent over character for character.
DeleteOne thing that I did yesterday that may have helped is that I revoked Quicken from my security profile in USAA and then went in and re-added it which does seem to generate a new clientUID. After this, I went and tried to add the USAA account in PocketSense using the newly assigned clientUID and then it worked. Please let me know if this works for you (or if you have already tried revoking and re-adding Quicken.
DeleteCapt_Ahab I didn't realize that a different clientUID was assigned. So this morning, Quicken had been automagically revoked from my permissions. I re-enabled it and used the NEW uid. But still, no joy. When it worked for you, what was the value for your appversion? I've seen some use QMOFX for both appid and appversion. Also, when it worked for you, were the account numbers automatically downloaded? or did you have to enter them manually after an initial error message?
DeleteI just now posted a new DEV version for testing. It's basically the same, but cleaned up a bit and made simpler. Specifically, I have deprecated the 3rd attempt for ofx downloads (leaving two), added userAgent='InetClntApp/3.0' on first call (previously omitted), and changed the default DTACCTUP value to 19900101. The userAgent and DTACCTUP values appear to be the current values used by Quicken, so I don't think it will break others. I'm leaving the optional site value for dtacctup and userAgent in case they're needed elsewhere.
ReplyDeleteIf any of these changes cause issues w/ other sites, I'll revert to the previous logic. All sites/accounts that I personally use work w/ these settings.
Hi Robert,
DeleteI tried this version but it seems to be generating errors that I haven't seen before.
D:\Pocket Sense>Getdata.py
Traceback (most recent call last):
File "D:\Pocket Sense\Getdata.py", line 54, in
import ofx, quotes, site_cfg, scrubber
File "D:\Pocket Sense\ofx.py", line 74, in
import getpass, scrubber, site_cfg, uuid
File "D:\Pocket Sense\scrubber.py", line 66, in
import site_cfg
File "D:\Pocket Sense\site_cfg.py", line 51, in
from rlib1 import *
File "D:\Pocket Sense\rlib1.py", line 21, in
from http.cookies import SimpleCookie
ImportError: No module named http.cookies
Hi Robert, I discovered that this morning, I was unable to pull transactions yet again from USAA. I did attempt run the latest script you posted but after it failed, I restored the first DEV version you posted but now I'm getting that "Request unsuccessful. Incapsula incident ID: 1337000090018244641-47402149679268425" error.
ReplyDeleteThe new version uses a package that's apparently not in the standard py distribution, so I'll need to manually implement that part or see if there's a standard (equivalent) package. When you say "first DEV version", I assume you had a local copy (?)... since the new DEV replaced the original in the link. Stay tuned on that part... but odd that the same script worked yesterday but not now, w/ presumably identical settings.
DeleteThis morning I noticed that my Quicken permissions in my security profile are no longer listed. I don't know whether they would have disappeared with a successful log in through Pocket Sense. Perhaps the credentials are only good for a certain period of time if there has not been a successful connection?
ReplyDeleteI have a quick update...after I waited 10 minutes and then used the new ClientUID after running the first DEV script, it started working again. I don't know if the ClientUID expires after a while or if running the new DEV script sent something to the ofx server and caused my IP to be blocked temporarily but I will keep testing this first version over the next few days to see if the connection will stop working if I leave things alone. That should isolate what caused the issue this morning. I should have tested the old script first before overwriting the python scripts and running it but I'll post the results in a couples days if it continues to work.
ReplyDeleteUSAA appears to be using Incapsula (Imperva) as a security gateway. My guess is that there's a delay between generating your clientUID and it being distributed/updated to the gateway server(s).
DeleteYeah...By the way, after setting up the account again, I had waited about two hours or so before running Getdata.py. When I ran it, it failed again. I went back into Setup.py and sure enough, it was back to pulling invalid ofx files and failing even though I had successfully re-added the accounts earlier. I'm wondering if maybe a new ClientUID is somehow being issued over time and the script would need to pull the new ClientUID each time or something.... I wish I knew more about how this whole setup works.
DeleteI doubt that the clientUID "expires", or if it does, it would be much longer. It's basically a "back door OAUTH", where you're generating an ID, PIN and token out-of-band. Citi has a similar process, except the clientUID is generated via ofx during first connect, but the user has to be on their "allow access to app" page at that moment. Once established, it's persistent.
DeleteThis does seem to be the case from what I can tell. After about an hour or so of just waiting (not revoking and re-adding Quicken access). The original DEV script seems to be working and pulling the account info. I'm a bit reticent to run the new BETA script again because my IP became blocked when I ran it.
DeleteSomething went wrong on my last post... Let's try this again...
ReplyDeleteHere's a new BETA for testing, (hopefully) using the right lib for Py 2.7.x.
Hi Robert, I was doing some reading on this hleofxquotes software that also pulls OFX data from sites. The instructions tell you to grab the clientID in the URL prior to accepting and then retrieving the AccessID and PIN. When I do this for Pocketsense, the script isn't able to pull data but it works just fine in hleofxquotes. At the moment, the IP seems to be blocked so I will have to play around with this script a bit later. So far, I can't say that the latest BETA is actually working but I just need to wait for the block to reset and try again later.
ReplyDeleteJust fyi, but the newest version is less likely to throw an ipsec flag, since it only retries one time (vs twice for the version you're using). I'm thinking that I'll suppress the backup attempt altogether, since I've restructured the request header and no longer needed.
DeleteHi Robert, I downloaded the script from the "new BETA" link. As I am running setup.py again, it seems that it will intermittently not be able to download the ofx file. Sometimes it will work and other times it doesn't. I haven't been able to figure out what causes the script to fail to download the ofx files. Since I updated it, so far my IP isn't getting blocked so this is good at least.
DeleteQuick update - I downloaded the hleofxquotes java tool and installed it on my home computer. Using that tool and querying for account info with the access Id and PIN and clientUID from the URL of the page with the "Allow" button, I am able to retrieve account info. I'm looking at the content of the OFX requests now to determine the delta from what Pocketsense is sending.
ReplyDeleteJohn, just to clarify, are you using the original DEV version from Robert or the newer BETA version he just released today? For some reason, the latest BETA version seems to cause USAA to block my IP temporarily but the DEV version seems to work. I will have to try this again to verify for sure the BETA version from today is causing my IP to get blocked.
DeleteSo...I copied the directory with the DEV version from the 15th of February and ran it again to see if it would work but now it's blocking my IP again. This was just after I ran it a few minutes prior. I'm wondering if maybe the USAA site blocks the IP if you send to many access requests in a short time. Rather than the script, maybe it's the frequency of making requests that causes the site to temporarily block your IP.
DeleteI was also able to use the hleofxquotes tool to download my account numbers and the transactions.
DeleteI used the client_id from the "Allow" page URL (I used lowercase and it worked).
I used my ID and PIN from that generated page (which have never changed through all removals/reauths).
It was able to retrieve my account numbers correctly. Putting in one (and later all) of my account numbers (they must be 10 digits, pre-pad with 0's if you have a shorter number; hleofxquotes does this correctly already if you download your account numbers) correctly pulled all transactions.
My sanitized, successful request (as reported in hleofxtools):
ReplyDeleteOFXHEADER:100
DATA:OFXSGML
VERSION:103
SECURITY:NONE
ENCODING:USASCII
CHARSET:1252
COMPRESSION:NONE
OLDFILEUID:NONE
NEWFILEUID:NONE
<OFX>
<SIGNONMSGSRQV1>
<SONRQ>
<DTCLIENT>20210217114849.216[-6:CST]
<USERID>myUserId
<USERPASS>myPin
<LANGUAGE>ENG
<FI>
<ORG>USAA Federal Savings Bank
<FID>67811
</FI>
<APPID>QMOFX
<APPVER>2300
<CLIENTUID>myClientIdFromTheAllowPage
</SONRQ>
</SIGNONMSGSRQV1>
<BANKMSGSRQV1>
<STMTTRNRQ>
<TRNUID>3108a63f-8059-45a2-a823-04ba09ebb2d7
<STMTRQ>
<BANKACCTFROM>
<BANKID>314074269
<ACCTID>my10DigitAcctNumber
<ACCTTYPE>CHECKING
</BANKACCTFROM>
<INCTRAN>
<DTSTART>20210118
<INCLUDE>Y
</INCTRAN>
</STMTRQ>
</STMTTRNRQ>
</BANKMSGSRQV1>
</OFX>
Andrew - Can you posted the sanitized version of your OFX account request. I see a few deltas in what is being sent from Pocketsense, but nothing major.
ReplyDeleteSure. Here's the HTTP header for both (account list request and transaction request), only difference was the content-length:
ReplyDelete=========================================
POST /casm/usaa/access.ofx HTTP/1.1
Content-Length: 667
Content-Type: application/x-ofx
Host: df3cx-services.1fsapi.com
Connection: Keep-Alive
User-Agent: InetClntApp/3.0
Accept-Encoding: gzip,deflate
=====================================
Here's the sanitized, successful "list accounts" request:
OFXHEADER:100
DATA:OFXSGML
VERSION:103
SECURITY:NONE
ENCODING:USASCII
CHARSET:1252
COMPRESSION:NONE
OLDFILEUID:NONE
NEWFILEUID:NONE
<OFX>
<SIGNONMSGSRQV1>
<SONRQ>
<DTCLIENT>20210217124459.836[-6:CST]
<USERID>myUserId
<USERPASS>myUserPIN
<LANGUAGE>ENG
<FI>
<ORG>USAA Federal Savings Bank
<FID>67811
</FI>
<APPID>QMOFX
<APPVER>2300
<CLIENTUID>myClientIDFromAllowPageLowerCase
</SONRQ>
</SIGNONMSGSRQV1>
<SIGNUPMSGSRQV1>
<ACCTINFOTRNRQ>
<TRNUID>3e0332cc-7e8e-43c4-8af6-6fd43e941261
<ACCTINFORQ>
<DTACCTUP>19700101000000
</ACCTINFORQ>
</ACCTINFOTRNRQ>
</SIGNUPMSGSRQV1>
</OFX>
The only things that jump out to me as being different from what Pocketsense is sending out is the DTCLIENT field and the DTACCTUP field. The TRNUID is going to be different from request to request, but the format is correct. The accept-encoding field is not present in the HTTP header that Pocketsense sends.
ReplyDeleteDTACCTUP is only used when listing accounts, and is included in the PS scripts when adding an account.
DeleteDTCLIENT is different format, and includes milliseconds + timezone suffix. These aren't required by OFX, but possibly something that the gateway checks (?).
Accept-encoding isn't required for OFX http headers, but perhaps (again) the gateway does?
Here's one other difference I've noted. The header length sent over by hleofxquotes has content length set to 575. The script sent over by Pocketsense has that value set to 608. I've modified my Pocketsense scripts to use the longer DTACCTUP and the DTCLIENT with addition of timezone info. I think the content values should be the same given that nothing else is different (i.e. username, password, clientuid, trnuid are all the same length. Can't figure out how the count for Pocketsense is 33 characters longer.
DeleteHere's the header from hleofxquotes tool:
POST /casm/usaa/access.ofx HTTP/1.1
Content-Length: 575 <<<<<<<==========
Content-Type: application/x-ofx
Host: df3cx-services.1fsapi.com
Connection: Keep-Alive
User-Agent: InetClntApp/3.0
Accept-Encoding: gzip,deflate
Here's the header from Pocketsense:
POST /casm/usaa/access.ofx HTTP/1.1
Content-Type: application/x-ofx
User-Agent: InetClntApp/3.0
Host: df3cx-services.1fsapi.com
Content-Length: 608 <<<<<<<==========
Connection: Keep-Alive
Quick follow-up. I can get the count to 575 if I remove one of the "\r\n" at the end of each field (i.e. replace "\r\n" with just "\n" or just "\r".
DeleteDoes the request include \r\n or just \n ? The PS scripts use \r\n ...
DeleteI can't tell what's actually sent over. I'm surmising based on the content length that it's not sending both. I'm looking at the output file that gets printed by hleofxquotes showing the request that was sent. The eol marker is about the only thing that can be different as the rest of the data is identical.
DeleteI've noticed that Pocketsense is using TLS v1.2 protocol and hleOfxQuotes tool is using TLS v1.3. Don't know how much difference there would be due to updated encryption protocol, but perhaps there is something. The only two deltas I see in the data string I'm transmitting via Pocketsense and the one I'm transmitting with the java tool is the TRNUID and the DTCLIENT. Everything other piece of data is identical.
DeleteI'm assuming that the hleofxquotes app is "open source", or you guys wouldn't be using it? Can someone point me to the code repo?
ReplyDeleteI haven't seen the source anywhere. The jar files are scattered around several software hosts, most recently at the BitBucket URL.
DeleteSure thing! Here is the link to GitHub where the project site is located: https://github.com/hleofxquotes/hleofxquotes and here is a page where the developer put a link to a bitbucket URL with the version that fixes USAA: https://infinitekind.tenderapp.com/discussions/online-banking/18262-usaa-using-hleofxquotes-to-download-ofx
ReplyDeleteJust look for the poster listed as hleofxquotes to find the post with the direct download link.
It looks like his main site for hleofxquotes doesn't seem to show a new release as of yet (maybe because it's in beta at the moment)
Unless I'm missing it, I only see compiled java files (byte code). No source...
DeleteAhh sorry about that...I reached out to the developer of hleofxquotes and he says he plans to release the open source on the latest code shortly.
DeleteRobert,
ReplyDeleteI have installed the new beta this morning, including the new sites.dat (*below), and after entering my ID and Password Setup.py disappears without a trace. Here are the error messages when I run **Getdata. Please advise, thanks.
**
invalid literal for int() with base 10" ': 103'
.\xfr\USAACHECKING...94.ofx is not recognized as an internal or external command..."
(Note that I didn't add all the numbers for security reasons)
*
url: https://df3cx-services.1fsapi.com/casm/usaa/access.ofx
fiorg: USAA Federal Savings Bank
fid: 67811
bankid: 314074269
appid: QMOFX
appver: 2300
ofxVer: 103
dtacctup: 19900101
userAgent: InetClntApp/3.0
clientUID: your_value_here
Also, what information should I use when Setup.py asks for Name(ID) and Password? Previously I would enter the same credentials I used to sign in to the usaa web page. Thanks
jmf
Not sure which version you're using, but I just now uploaded another DEV version for testing. It's similiar to the last beta I posted, but using \n separation rather than \r\n. I tested this version w/ the sites I use and it works correctly w/ those.
DeleteAs for the Name / Password, from what I've gathered that's the ID and PIN assigned by USAA via the "enable connection" form. I don't use USAA, so just going by what others are posting.
DeleteRegarding the error above, that actually looks like you may have an invalid character in the site name that's throwing a file load error. Try renaming the site w/ no spaces or special chars (e.g., USAA-TEST would be fine). The scripts attempt to cleanup invalid chars from site names when creating files, but there's possibly a character in there that's missed in the filter but not allowed in windows file names.
p.s. If you rename the site, you'll have to delete & recreate the account in Setup using the new name.
DeleteI posted this as a reply above, but want to separate it as a new post. I uploaded another DEV version for testing. It's similar to the last beta I posted, but using \n line separation in the ofx-message rather than \r\n. I hasn't mattered for any other ofx service, but worth checking.
ReplyDeleteOops... here's the link to the latest DEV version.
DeleteHi Robert, I tried the DEV version and the test run on the account is still failing. Checking with the hleOfxQuotes tool, I can see that my IP isn't getting blocked at least.
DeleteI forgot to mention that the dev of hleOfxQuotes did provide some documentation on how the program communicates with USAA. Here is the link: https://bitbucket.org/hleofxquotesteam/hleofxquotes/wiki/USAA_Background_Notes
DeleteI've made it a little bit further in troubleshooting the USAA problem. I did an account request using the hleOfxQuotes tool and successfully received a list of accounts. I copied the DTCLIENT data and the TRNUID data from that request and inserted that data into the Pocketsense request and voila I finally received account data using Pocketsense. I suspect this means that the TRNUID is somehow derived from or tied to the DTCLIENT string.
ReplyDeleteDid you test with only the DTCLIENT field changed? TRNUID should be a unique value for each request, but clientUID is a static value across connections.
DeleteAlso... want to make sure we're using the same code, and that tests were w/ the latest DEV version.
DeleteThe DTCLIENT field has the following value: 20210217174516.169[-6:CST]
DeleteThe TRNUID field has the following value: f638e084-43a1-47d2-9a8d-ebe91441d664.
I'm using scripts that I've edited. These would probably be close to your latest dev version. I'm sending just a CR as a separator rather than a CRLF pair. I'm going to try to use just the time value from hleOfxQuotes tool with Pocketsense generated UUID for TRNUID and see if that fails.
Robert, re "back door OAUTH", any chance any of the new coding applying to Schwab dwonloads?
ReplyDeleteIs Schwab possible w/o having to sign up as a "3rd party company", w/ approvals & legal forms?
DeleteHere's another piece of the puzzle. The TRNUID that's sent as part of an account query request is also echoed back in a successful response. Here's the TRNUID field in the request 314fb85b-f74d-405d-85bd-813f0c4472b1 and here's the TRNUID in the response 314fb85b-f74d-405d-85bd-813f0c4472b1.
ReplyDeleteTo this point, the only successful response I've received to an account query request in Pocketsense is when I copied values for DTCLIENT and TRNUID from a successful hleOfxQuotes account query request and injected them into a Pocketsense request. Every other attempt has failed.