The DEV post below exceeded the 200 comment limit, so this is a continuation for easier commenting. See post below for info and link to latest DEV version. It's been in use long enough that I need to just push it along,
- Robert
The DEV post below exceeded the 200 comment limit, so this is a continuation for easier commenting. See post below for info and link to latest DEV version. It's been in use long enough that I need to just push it along,
- Robert
New DEV version posted to fix recent Citi issue. Let me know if it breaks any other connections. If so, I'll modify how it works to be more flexible. -Robert
ReplyDeletep.s. If your Citi connection still doesn't work, re-add the account while using the Citi "manage desktop apps" feature on their site.
DeleteWorked great for Citicard.
DeleteJust copied the contents of the *.zip file over the contents of that folder.
I also did a fc. Cool.
I installed the 2021-Aug-18 DEV version and am now getting this message:
DeleteWe've recently updated our online banking experience. To continue using this app, you'll need to relink the app to your Citi account. You can complete this at https://citi.com/datamanagement. As a reminder, only credit card accounts can be linked.
However, re-linking is not working. I am using the same parameters that were previously working:
SiteName : Costco Citibank Visa
AcctType : CCSTMT
fiorg : Citigroup
fid : 24909
url : https://mobilesoa.citi.com/CitiOFXInterface
bankid :
brokerid :
ofxVer : 103
appid :
appver :
mininterval :
timeOffset :
delay :
skipZeroTrans:
userAgent :
dtAcctUp :
clientUID :
Any ideas what the problem is now?
Paul White
Did you try to add the account while the Citi desktop app is in "add access" mode ?
DeleteNot sure if it matters, but I had appver: 2700 for testing. I just now changed the default appVer to 2700, as I think it's ~ time.
DeleteRobert: I am doing what I understand I should do to add the account while the Citi desktop app is in "add access" mode. I do this by going to https://citi.com/datamanagement, logging in with my Citi user ID and password, and then clicking on Continue when I see this message:
DeleteAllow access?
After agreeing to set up your connection, you will have ten minutes to complete the process by signing on to your financial management app or website and viewing your Citi account information.
I then run GetData, which is what I did in the past to get this working.
I tried it with appver : 2700 but got the same result.
Rather than going straight to getdata at the last step, run setup and add the account again while the Citi "allow access" listener is enabled. That should re-connect the app by getting a new key.
DeleteRobert: Do I need to do anything about the connect.key file Cal mentioned? I don't understand what that does.
DeletePaul
No. Adding the account again will reset the connection key.
DeleteRobert: I can't get this to work. I have used Setup to delete and add the account multiple times and even renamed it in Sites.dat to make sure that it was really gone. I have tried deleting the connect.key file to see if that makes a difference, but it does not seem to.
DeleteWhen I enter the user id and password, I receive this message:
"An error occurred requesting accounts from the site. Please check username and password."
That seems to indicate that the log in is not working for reasons I cannot figure out. Maybe Citi is blocking this but not telling me? I keep getting this:
"We've recently updated our online banking experience. To continue using this app, you'll need to relink the app to your Citi account. You can complete this at https://citi.com/datamanagement. As a reminder, only credit card accounts can be linked."
Any ideas?
Paul
Robert: To clarify, I am getting the error message when I use Setup to enter the user id and password.
DeleteThat message indicates that the scripts are connecting to the server correctly but that the computer isn't being recognized as "authorized". Did you delete the existing "connected applications" on the Citi site before trying to add it again?
DeleteI had not deleted existing "connected applications" on the Citi site so I tried that and now it is working. Thanks for your help.
DeletePaul
Worked fine for me, Robert. For my AT&T card. Also did not seem to break anything else I use. Thank you so much.
DeleteI got curious, and went to https://citi.com/datamanagement
DeleteI got a pop-up that said
Allow Access?
After agreeing to set up your connection, you will have ten minutes to complete the process by signing on to your financial management app or website and viewing your Citi account information.
----there were 2 choices, and I clicked the one on the right, which may have been "Cancel".
That got me to
------------------------------------------------------------
https://online.citi.com/US/ag/authorizedapps/manageaccess
Sharing Your Account Information
Based on your direction, we’re giving the applications and websites listed below access to your Citi account information. You can review and make changes here at any time. Just a reminder—resetting your User ID or password won’t do the trick. However, if you’ve shared your User ID and password directly with an application or website that isn’t listed here, you can revoke their access by changing your log-in credentials.
[and then a diffrently formatted section that said]
You have not authorized any apps or website access to your Citi Information
------SO, Nothing in the list!--------------
Yet somehow Citicard download has worked with Microsoft Money direct access and then PocketSense for me since 1999. Thanks Robert!
Cal, I am using a Costco Citibank Visa card and had about 8 entries in the list that was empty for you. I believe the oldest one was from 2017. These were apparently from various times that I needed to re-link for PocketSense downloads. After I deleted all of the entries I was able to re-link and get the PocketSense downloads working, and there is now 1 new entry.
DeleteIt sounds like there is a different way to make this work, or maybe it works differently for some credit cards. In any case, this is working for me again.
Paul White
Paul: was your connect key changed when you went to Citicard?
ReplyDeleteIs your new DEV code in a different folder? The connect keys are in connect.key file. So if copying credentials over to a new folder, copy that file as well as ofx_config.cfg which has the other password stuff in it.
My Citicard just somehow created its own connect key originally, and PocketSense integrated it for me. When you go to the Citicard website, do you get a string that you copy over and enter into setup.py?
Your sites.dat descriptor matches mine, other than what we chose as SiteName.
Cal: I extract the PocketSense zip file into a new folder, copy in the xfr and import folders, copy in my ofx_config.cfg file, and copy in my sites.dat file after merging in any sites.template changes.
DeleteI have never done anything about the connect.key file and was not aware of it until you mentioned it. However, I see that there is a connect.key file in all the PocketSense folders I have used since 2018.
I use Microsoft Money on 2 different PCs and install new versions of PocketSense the same way on both PCs with a fresh PocketSense folder and same copies. The connect.key folder is not getting copied between the two PCs.
When I go to the Citi website, I do what I described in my post to Robert. I do not get a string that I copy into Setup. I am not using Setup in trying to get this working because I generally just need to run Getdata.
Paul, thanks. I guess that action on the Citi site says to accept whatever comes in in the next several minutes with the correct login and password.
ReplyDeleteIf you add copying connect.key to your procedure for a next version, you won't have to visit the web page. On this version I did not create a new folder, but just used the same PS_dev folder I had been using.
If you go to http://pocketsense.blogspot.com/2021/02/dev-testing.html and Ctrl+F for connect.key, you will see where I first became aware of connect.key. Doing a similar search on earlier threads will show others had become aware and dealt with it before I did.
Paul, I doubt that you will get by with zero entries in your list, because I think mine was somehow grandfathered in. But I think you can keep just one. Every time you copy ofx_config.cfg, also copy connect.key.
ReplyDeleteStill, I think they have my connect key memorized despite not showing in the list.
Thanks for the insight. I wonder how you had 8 entries but a 9th would not get entered. Maybe they put a limit for the size of that list.
Cal (or other experts),
ReplyDeleteCitibank doesn’t seem to use proper ofx programming for their Bank Statement QFX files (their credit card qfx is fine though).
Here is how Citibank reports:
{BANKTRANLIST}
{DTSTART}20210904
{DTEND}20170204210000
Which is incorrect, and Money uses the 2017 date as the “Last Downloaded” entry. It also messes up the starting balance entries.
I created a sites.dat entry for Citibank_import and was hoping someone could help me with a scrub routine to fix the DTSTART and DTEND data.
Pocketsense scrubber has a routine _scrubDTSTART(ofx), but that’s for adding a missing DTEND.
If I switch the data as follows, then Money processes the data properly:
{BANKTRANLIST}
{DTSTART}20170204210000
{DTEND}20210904151536 (I used the date from {DTSERVER})
Cal, I’m already using your:
from ps_subs import make_substitutions #added by ameridan ****
if 'CITI.COM' in siteURL: ofx = make_substitutions(ofx) #added by ameridan ****
if you want to use the same logic.
(It makes it hard to update my Pocketsense scripts with all of these independent scrub routines inserted everywhere though)
-ameridan
SiteName : CITIBANK_IMPORT
DeleteAcctType : BASTMT
fid : 2102
fiorg : Citibank
url :
bankid : 271070801
brokerid :
appid :
appver :
mininterval :
timeOffset :
ofxVer :
(there will be no siteURL)
I was unaware of Money's use of the {DTEND} field.
DeleteJust to re-iterate for those reading is this is not about the well-behaved Citi credit card.I just did a Citibank web download to see what I see. I confirm your observations. As usual, before downloading I set a custom date range to avoid overlapping with previously download transactions. However looking at the FITID, I suspec that may no longer be necessary. Their using a different FITID when later downloading the previous transactions was irritating, and that may be fixed.
Looks to me as they interchanged the {DTSTART} and {DTEND}fields. Another fix might be to test, and if DTSTART is later than DTEND, then swap the fields.
So anyway, I should be able to write a Citibank scrubber doing what you ask for. I tend to only do that when I happen to be inspired vs dull.
Note that since some use both bank and credit card, then checking which scrubber to invoke would be based on the ("CCSTMT" in accType)or ("BASTMT" in accType)
My latest DEV scrubbing defines, and could use
#ofx=_Subsitute_first_ACCTID_chars_ofOFX_with(ofx,"XXXXXXXXXXXX")
instead of the make_substitutions(ofx) function. I thought that had the advantage of not needing to include anybody's account numbers in the source.
But that is actually commented out in my code, since Citicard stopped that XXXXXXXXXXXX thing.
I have thought of reviving my ps_subs idea, but I did the all-in-one file last iteration because it was easy, and I was not sure anybody would be using this. I really did not get comments about the functions. Separate scrubbers per FI and a ps_subs with common elements would seem more elegant.
Having the translation data encoded into a hard-coded tuple was pretty hokey. I am thinking that passing the data as parameters, and if you want three things, for example, possibly substituted, call it 3 times.
Cal, Any luck on the DTEND scrub. Might be easiest to just DTEND="now", like Robert does it.
ReplyDeleteI just downloaded a credit card QFX from the web, and observed that they still redact ACCTID, contrary to your statement "But that is actually commented out in my code, since Citicard stopped that XXXXXXXXXXXX thing.".
I have it functioning. What form would you like it in?
DeleteA function where you would do
ofx=put_date_from_DTSERVER_into_DTEND(ofx)
??
Cal, I don't really know what you are asking. I’m already using your code insertions:
Deletefrom ps_subs import make_substitutions
if 'CITI.COM' in siteURL: ofx = make_substitutions(ofx)
and it makes sense to do something similar, so I can edit updates easier.
That is what I was wondering. So you are not using the DEV version. I run both ways separately, currently
DeleteSo you are not using the latest DEV versions. I got to trying to integrate this into code for the DEV version, and I found I did not know how to make that work. The import function cannot be invoked by the test function of setup.py, so only getdata.py.
I think you may be saying that you would like a copy of ps_subs with a function called
def put_date_from_DTSERVER_into_DTEND(ofx)
=================================
Let ~ represent indent.
definition can be in scrubber or added to ps_subs.py
=============================
def _read_DTSERVER(ofx): #returns date from DTSERVER for further use
~found = re.search('.*<DTSERVER>\s*(\d\d\d\d\d\d\d\d).*', ofx).group(1)
~found=found+"1200" #avoid any timezone problems
~print type(found)
~return found #as string to be used as date
=================================
#and in your citibank scrubber, have
~ofx= re.sub( "<DTEND>\s*\S+[^\s<]","<DTEND>"+_read_DTSERVER(ofx) , ofx)
Oh, I didn't realize things have changed. I'm using DEV, but not the newest. Ver: 16-Sep-2019 DEV.
DeleteI'll see if I can get this working.
Cal,
DeleteThank you for your help. I couldn't get it to work, so I'll just do it manually until I close the Citibank accounts. Citi offered me a handsome bonus to open up accounts, but because of the poor customer service, after I get my bonus, I'll close them again. Now I see why they have to bribe people to open accounts at Citibank.
An FYI,
DeleteI manually deleted the DTEND line, so that the generic scrubber would add DTEND along with the correct date. The editor creates an ofx.bak file, which he import editor also loads, and natually that file still has the bad date, so it overrides the first file. So now I also have to delete the .bak file after making any manual changes, prior to importing.
I made a revised scrubber. In the DEV code, scrubber.py is provided with the PocketSense download, and custom scrublets can be run. If any file of the form scrub*_.py is present in the folder, it gets run.
DeleteFile scrub_cl5.zip has the scrub_cl5.py plus a little text file. Should be available at
https://drive.google.com/file/d/1Km0ja3HpA7YErn3gLi1ivn--bUusTefm/view?usp=sharing
The plan was that that way there could be a bunch of provider-specific scrubbers present. I expect that is how it will work out. It makes sense. However for my convenience I put all of my stuff into a do-it-all scrubber for easier experimenting. I have made a revision of what I had provided before, which was scrub_cl5.py. If using that, rename it to something like scrub_cl5.py.old, or anything that does not match the pattern.
I had not experiment with the import folder previously other than to see stuff placed in import folder got sent on. I did not check out scrubbing one.
I worked out code to modify DTEND to be a copy of DTSERVER but when I tried making a scrubber to use that I was not able to get that scrubber section to be invoked.
What I have hit on is a function that solves the conundrum
ofx=check_order_DTSTART_DTEND(ofx)
It does nothing unless the DTEND date is not later than the DTSTART. And if it finds that, it interchanges the two values. Seems harmless, so I am doing it for all regardless of testing which FI it is.
What this new code does not have is the account number substitution thing. What I do is for citicard or citibank, it matches the current obfuscation, so if they start using the whole number at some point, money won't notice. The problem with swapping is that it has to be modified with your account number, so that makes sharing harder.
When you use new PS versions. I like to put everything into a new folder. It is *important* to copy some files over into the new folder from the working folder:
ofx_config.cfg and connect.key, plus sites.dat of course.
I should have put my sites.dat into the zip file, but it would not be something that you need.
Wow - that took awhile. Join GitHub, download new Python 2.7.18 package, install, get rid of all my test scrub.py routines, copy new DEV files, and your scrublets. So I'm caught up to you now Cal, and the scripts are working again. I'll test the Citibank at a later time. Thank you for your assistance!
DeleteCal, it works!
DeleteHere is a section of the feedback (note that the citibank scrubber runs for every bank):
PNC : xxxxxxx: Getting records since: 20210904
+siteURL:~:HTTPS://WWW.OASIS.CFREE.COM/4501.OFXGP
+accType:~:BASTMT
+An error occurred when processing scrub module: scrub_cl5
'NoneType' object has no attribute 'group'
PNC IL ALT : xxxxxxx: Getting records since: 20210810
+Scrubber: Null time values updated.
+Scrubber: tags removed. Not supported by Money.
+siteURL:~:HTTPS://WWW.OASIS.CFREE.COM/4501.OFXGP
+accType:~:BASTMT
qq1
+ran _prettyOFX(ofx)
qq1
+Citibank scrubber ran
Searching .\import\ for statements to import
Importing CHK_3409_CURRENT_VIEW.qfx
Matched import file to site *CITIBANK_IMPORT*
+Scrubber: Null time values updated.
+siteURL:~:HTTPS://MOBILESOA.CITI.COM/CITIOFXINTERFACE
+accType:~:BASTMT
qq1
swapping dates for DTSTART and DTEND
+ran _prettyOFX(ofx)
qq1
swapping dates for DTSTART and DTEND
+Citibank scrubber ran
Finished downloading data
Combined OFX created: .\xfr\combined586138.ofx
Sending statement(s) to Money...
New scrub_cl5A.zip contains
Deletescrub_cl5A.py scrub_cl5A.txt scrub_cl5X.py.debug cl5A_sites.dat.example
zip available at https://drive.google.com/file/d/1Rs_iFKHyq17yjTxHw-xfs5bFLxslULUJ/view?usp=sharing
scrub_cl5A.txt is a file with info like below, plus thoughts on how to implement substitution in a better way. It is not a howto.
Multiple inappropriate running of Citibank import scrubber addressed by looking for stuff in ofx rather than in parameters passed in.
Cleaned up superfluous temporary debugging prints but kept them in scrub_cl5X.py.debug for possible reuse.
cl5A_sites.dat.example is a copy of my sites.dat.
Earlier scrub_cl*.py should be renamed or deleted so as to not run two of them.
The latest prettyOFX works very nicely! I hadn't realized that Robert's latest DEV accommodated user-added scrubs, making it very convenient for Pocketsense upgrades without the need to edit his program files.
DeleteHi Robert,
ReplyDeleteHave you ever considered hosting the PocketSense code in some kind of Git repository (such as GitHub or GitLab)? It would make it easier for developer types to pull in your changes or to contribute their own. You would still be able to make a ZIP version available to those who wish to download and install that way.
Thanks,
Andrew
Andrew: I use git for local tracking, but haven't considered a public version. I track everything, including stuff I wouldn't want to share, and the zip files have been a ~ simple way to provide the pkg w/o needing an account somewhere. Perhaps an in-between option would be to build out a git history from the zip archive I've maintained, and make it public for cloning? I guess there's no reason to back very far, though? Something to consider...
DeleteHi Robert, with care, you could just publish the correct changes to a public repository to make it easier for others to download and contribute. A safer alternative would be to maintain two local repositories and then manually copy files from your "development" repo to the "public" repo to avoid any accidental cross-contamination (usernames/passwords/keys/etc) while maintaining git's features on your development repo. It's also fine to just start from somewhere and upload the latest DEV version as the initial commit.
DeleteI'm having an issue with Chase credit card imports. I'm using the pre-dev version of PocketSense, as well as MS Money 2006. Chase is set as ofxVer 103 in sites.dat, which downloads properly but fails when importing into Money (Money says it's a corrupt file). If I change within the downloaded ofx file to ofxVer 102, then the download imports correctly by Money. I tried changing the version to 102 in sites.dat, but then I can't download anything as Chase gives the error.
ReplyDeleteWould this scrubber feature enable me to change the version from 103 to 102 to enable the import, or is there another fix that I should have enabled previously on either MS Money or PocketSense? Or would updating to Money Sunset fix this issue? I tried installing the DEV version to see if there was something updated, but it is failing and I may have to reinstall python as mentioned by Cal earlier.
I would have no idea of how to use the scrubber feature if that's what I need to do...
Thanks,
Joe H
Joe: I could make a scrubber that changes all instances of "VERSION:103" to "VERSION:102".
DeleteThis could be limited to only process your Chase ofx files, or to run on all files.
To do that, you would need to change over to the recent DEV version, and to put the custom file into the folder with the new stuff.
If you are going to try this, I would propose to make a howto to describe how to do that. Synopsis of my suggestion would be to create a different folder to unzip the DEV version into, a short list of files that you would copy from your current PocketSense folder to the new one. Alternatively, maybe copy the existing folder named PS_now to PS_old2021 in case you want to go back. Then unzip the DEV files to add to PS_now and overwrite many existing files. That avoids having to copy that short list of files from your existing folder.
Which of those methods appeals to you more if you were going to try DEV?
Anyway, I would provide a file, prospectively named "scrub_change_VERSION_from_103_to_102.py"
No change would be made if the ofx did not contain VERSION:103
This file gets added to the PS folder you will use. Then the DEV software looks for any files named scrub_*.py and runs it/them.
I expect PocketSense is getting the OFX file for you, rather than you using web. Scrubbing can work either way, but knowing that might affect the writing of the scrublet if trying to make the scrubbing selective.
Hi Cal,
DeleteSomehow my weeks ago reply didn't happen - no doubt user error in not hitting publish.
I finally decided to bite the bullet and upgrade to Sunset which removed the issue. I stayed with 2005 only because of Ultrasoft Money and hoping to get it to work again on the Palm app on Android. After 3 years not trying, it was goofy to hold out!
Thanks for the offer!
Joe
Joe,
ReplyDeleteI don't believe Chase requires scrubbing, so in theory, your version of Pocketsense should be fine. ofx version is 103 though, so my guess is that Money 2006 isn't processing quite the same as Money 2008 (Sunset).
See if this helps you: https://microsoftmoneyoffline.wordpress.com/2018/01/07/chase-ofx-downloads/#comment-14033
Robert,
ReplyDeleteI discovered a coding error that I made in the T. Rowe Price scrubber routine from July 2018. It is a destructive error that causes valid reinvested transactions to get deleted. I have posted an updated scrubber marked Version 2 that corrects this mistake.
Meanwhile, T. Rowe Price has corrected their OFX formatting for div/cg transactions that are paid out. Therefore, there is really no reason to have this scrubber anymore. The imperative thing is to make sure no one has the old scrubber by putting a new scrubber on your official release page. It seems there are two options:
1) Post this updated scrubber in case the original OFX formatting problem reappears.
2) Post a "do nothing" scrubber that will overwrite earlier versions.
Here is the link to an updated scrubber as per 1) above:
https://1drv.ms/u/s!AmLW6sj6wM5Q3h6BbntGCAdjjhvj?e=5HGomZ
If you want me to create a "do nothing" version as per 2) above in order to streamline execution, please let me know.
I apologize for my mistake and the impact on you and the community.
David
No problem, and thanks for reporting! I'll include a null scrub file of the same name to replace the existing one, and remove it later on.
DeleteI was making a Fidelity-only scrubber for possible inclusion for general use in the DEV code. I named it scrub_fidelity.py and it is available at https://drive.google.com/file/d/12bh2whCa4cvkxRSTFyxYSlkouCjraeay/view?usp=sharing
DeleteIt is a good one IMO.
Odd thing... During testing I got this:
===================================
+An error occurred when processing scrub module: scrub_fidelit
No module named scrub_fidelit
Download completed successfully
=============================================
When I renamed scrub_fidelity.py to be scrub_fidel.py, it ran without errors.
I expect it should have worked while called scrub_fidel.py. Some one-time goblin? Can't be the number of characters is a problem.
Having login problems.
Cal Learner.
I expect this post to fail, but here goes.
ReplyDeleteSHA-1 was 76f8e3367eba608410cc427cfd2052d0290e93ed for scrub_fidelity.py that I posted.
On my last couple of script runs I am getting an invalid OFX statement error for T. Rowe Price. Is there a problem with the T. Rowe script?
ReplyDeleteRegarding my post about T. Rowe Price I see a post from 9/15 about T. Rowe Price about a scrubber routine error but I don't undertand the comment. Can someone explain and is it related to the problem I am having?
ReplyDeleteIf you run setup.py you will see
DeletePocketSense OFX Download Python Scripts, Ver:
with a version that follows. Knowing that version could give insight.
David Casada on September 15, 2021 said that if you delete scrub_troweprice.py that would be good. See if deleting that file helps. I don't use that FI myself.
Regarding the two questions above:
Delete1) Question: Is there a problem with T. Rowe Price script? Answer: There is a problem accessing T. Rowe Price with these scripts for about the last month. Web site ofxhome.com shows TRP has failed since mid September. I have talked to TRP tech support and scanned various web blogs for resolution. What I know is that TRP implemented a new OFX server with this address: https://api.public.troweprice.com/ffs-ofx-direct/service/ofx-request
However, changing that url in sites.dat does not resolve the connection problem. I got so desperate that I bought a copy of Quicken to see if it could connect, and the answer is that Quicken connects to TRP seamlessly. So my assumption is that TRP and Quicken made an unpublished interface change. Note that USAA.com did this too within the last year, and eventually the community figured out how to make the scripts work. So I am hopeful we can resolve this too. Unfortunately there aren't very many TRP users so the community activity level isn't high. I will post a solution on PocketSense if I can find out what's going on at TRP.
2) Question: Is there a T. Rowe Price scrubber routine error affecting the connection to TRP? Answer: Yes, there is a scrubber error, but no it has no impact at all in connecting to TRP. You can safely delete the scrubber as it is no longer required.
David: Ofx Home unfortunately has become read-only for a year, and then will go dark. It does not have updated info. We will miss it.
Deletehttp://www.ofxhome.com/ofxforum/viewtopic.php?id=49843/ofxforum/viewtopic.php?id=49843
David: While searching around I found the phrase "packet sniffer on a PC running Quicken". I don't know what it would take to packet sniff the Q request packet, but maybe Christopher Singley could get you going. https://readthedocs.org/projects/ofxtools/downloads/pdf/latest/
DeleteHi Cal,
DeleteI too noticed that OfxHome is going dark early 2022 unless someone takes over for the retiring creator/maintainer. In the OfxHome blog a fellow who wrote MyMoney (an fairly sophisticated application for managing your personal finances, see https://github.com/clovett/MyMoney.Net) offered to take over OfxHome, but I'm not sure if the handoff has been agreed. Hopefully, someone keeps OfxHome alive.
As for the Quicken packet sniffer, I did look into that and eventually found a tool called WireShark that looked promising. But to be honest, it was way over my head trying to figure out how to use it. Alternatively, I installed Quicken which successfully connects to T. Rowe Price. Then I searched my registry and hard drive .ini files looking for any Quicken configuration information that might give me a clue about how they are connecting. Alas, I didn't find anything useful. I keep scanning the internet for people working on the T. Rowe Price problem, but see very little activity.
Thanks for you tips on this problem, and for all your work on the PocketSense scrubbers.
I just got my T Rowe Price download to work for the first time in a couple of months. This is my setup entry
DeleteSiteName : T ROWE PRICE
AcctType : INVSTMT #investment
fiorg : T. Rowe Price
fid : 40109
url : https://api.public.troweprice.com/ffs-ofx-direct/service/ofx-request
bankid :
brokerid : troweprice.com
ofxVer : 102
appid : QWIN
appver : 2700
mininterval:
timeOffset :
I tried the sites.dat configuration above for T. Rowe Price and continued to have problems connecting since Sept 15, 2021. Finally, I found that I could review the Quicken log files and see what QFX (OFX) configuration they use for Direct Connect. Based on that analysis I found that using this setup sites.dat works for me using the PocketSense scripts:
DeleteSiteName : T. Rowe Price
AcctType : INVSTMT #investment
fiorg : T. Rowe Price
fid :
url : https://api.public.troweprice.com/ffs-ofx-direct/service/ofx-request
bankid :
brokerid : troweprice.com
ofxVer : 102
appid : QWIN
appver : 2700
mininterval :
timeOffset :
delay :
skipZeroTrans:
userAgent :
dtAcctUp : 19900101
clientUID :
Happy New Year 2022!
Spoiler alert: I am fine now. I had a scare this morning. My scheduled Getdata.py fetching did not happen. The batch file reference to c:\python27\python.exe had been stymied. The file did not exist, and that part is still a mystery. I did a virus scan, which came up clean.
ReplyDeleteI run this on my Windows 7 32-bit machine, and that limits what versions of Python 2.7.1x are available. But I did find msi files for 2.7.15 Python.org version. So I ran that. Got an error "There is a problem with this Windows Installer package. A program required for this install to complete could not be run. Contact your ...."
Well dang. Did the installer get messed up somehow? I downloaded python.org and python-2.7.13.msig-2.7.14.msi and tried each of those with the same result. Dang... looked up lots of stuff. Retried stuff. Turns out that that was a less-than-helpful error message IMO.
So after a while, I decided to uninstall my python 2.7 before proceeding. That gave a useful error message that it could not find python-2.7.12.msi to do the uninstall. I suddenly had hope. I downloaded python-2.7.12.msi from python.org. Tried the uninstall again. This time there was a choice presented to do a repair install. I chose repair. Joy! Everything works.
Incidentally, trying to get a suitable package from ActiveState was really confusing. I did have the GitHub account. It was asking if I wanted to fork, and I was finding that way confusing. I used to get packages from ActiveState no problem. Maybe I am not as adept today.
If I were to install Python fresh today for PocketSense to run on 32-bit Windows, I would go with python.org-2.7.15.msi from Python.org. But since 2.7.12 is working so well for me, I will stick with that for now. Going through https://www.python.org/downloads/windows/ to find a 2.7.11 and above that runs on 32 bit windows is a little tedious, but not bad. The Python version 3 is being presented a lot more prominently.
For a new person wanting to run PocketSense today with the newest Python 2.7, I would suggest downloading and installing https://www.python.org/ftp/python/2.7.18/python-2.7.18.msi which should work with 32-bit or 64-bit windows. If 64-bit, you could use https://www.python.org/ftp/python/2.7.18/python-2.7.18.amd64.msi I have not run those.
If you want the one I am using successfully, it would be https://www.python.org/ftp/python/2.7.11/python-2.7.11.msi
I expect installing Python 2.7.11+ could be a problem for a typical new person, and suggesting specific links could make that easier.
Fidelity "CORE" Positions (sweep) Scrubbing -- do you use FCASH?
ReplyDeleteMajor question: what is the CUSIP for FCASH?
I have a Fidelity Investments scrubber, I have shared a version, and I want to improve it. One feature that works for me is that I remove Buys and Sells of my core (sweep) fund, so that the cash balance in Money reflects my cash and cash equivalent. I hard code the CUSIP into my scrubber.
My plan is to make this scrubber more useful to others without requiring changes to the Python file. My thought is to have an ordered list of known Fidelity sweep funds, and go through the list and identify which is present in the OFX file. Then remove Buys and Sells of that fund. If more than one occurs, my current thought is to maybe pick the first on the list. Alternatively I could make sure there is only one those present in the OFX, and if there is more than one, require intervention.
Some have a "core" holding called FCASH. My question on that is does FCASH show a CUSIP in the file for Buys and Sells, or are maybe no Buy or Sell transactions in the file if you use FCASH?
My list will include
Fund CUSIP
FCASH unknown
SPAXX 31617H102
FZFXX 316341304
FDRXX 316067107
Any other comments or questions on a Fidelity scrubber?
I made up my own for Schwab Cash Reserves™ = CA$H, which you're welcome to use.
ReplyDeleteSo to drag things back to the previous USAA comments as I'm apparently still an idiot.
ReplyDeleteSites.dat entry:
SiteName : USAA
AcctType : BASTMT
fiorg : USAA Federal Savings Bank
fid : 67811
url : https://df3cx-services.1fsapi.com/casm/usaa/access.ofx
bankid : 314074269
brokerid :
appid : QMOFX
appver : 2300
mininterval:
timeOffset :
ofxVer : 103
dtacctup : 19900101
userAgent : InetClntApp/3.0
clientUID : (GUID from the Deny/Allow page)
Username/password are the AccessID and AccessPIN from the Allow page.
Prompted when I put them in Setup:
An error occurred requesting accounts from the site. Please check username and password.
Asks for my account number, I provide it, and then the downloaded OFX has the same Incapsula error.
Any ideas where I've gone wrong?
Hi Derek,
DeleteYour sites.dat entry looks correct. Which version of PocketSense are you using (it's on the first line of text that comes up when you launch GetData.py)? Your clientUID should be of the form: 8-4-4-4-12 hexadecimal digits (0-9a-f).
On the USAA website, go to your profile and either "Login and Security" or "Security" to get to the "Security" tab. You should see "Quicken" listed in the "Connected Apps" section.
PocketSense OFX Download Python Scripts, Ver: 2021-Sep-15 DEV
DeleteConfirmed that Login and Security on the USAA page does list Quicken (after yesterday's troubleshooting I got a stack of "Quicken connected to your USAA Account" emails). The clientUID is indeed in GUID format, lowercase as it was in the URL.
Cal - I've tried both ofxVer 102 and 103 and it fails either way.
I've also tried renaming connect.key to see if something in there was creating an issue; it recreated the file but still didn't work.
Am I going at this wrong - should I run through setup and pretend everything is good (skip the test) in hopes that USAA will decide it's OK on the next run?
Well, going through the add process and waiting for the "Quicken connected" email did not help - I'm still Incapsula.
DeleteDerek, I don't find Incapsula in any dictionary. I infer you mean that the OFX file that comes into the XFR folder has an error indication rather than useful data.
DeleteHahhaha - yeah, sorry; the OFX error from USAA when it's unheppy is "script src="/_Incapsula_Resource?"
DeleteAndrew and derek:
ReplyDelete1.You can also get the version by running Setup.py, and that will not try to access anything, unless you choose Test.
2. I presume that the second occurance of USAA (change log item 3) on http://pocketsense.blogspot.com/2021/02/dev-testing.html#comment-form meant to say "This option requires ofxVer=103". I was at first wondering if derek should change ofxVer, but no.
It appears that the download link is now broken. Any chance it could get updated? Thanks!
ReplyDeleteDavid: https://drive.google.com/file/d/1mzo8NhEtbUgiOdBmVOv7WJeqI4B-1tAS/view just worked for me when I tested.
DeleteIs that the link you are talking about?
Hi Cal,
DeleteThe link that you provided does indeed work but it seems that the link to d/l the script from the Pocketsense website errors out.
403 "We're sorry, but you do not have access to this page. That's all we know."
-Kevin N.
I fixed the main link, but be aware that it's to the release (old) version. The "dev" link at the top of this post is highly recommended. I need to update the main link to this one, but it means adding a few more instructions and reviewing things, and just haven't had time lately.
DeleteThank you Robert for fixing the link.
DeleteI was introducing a new user to Pocketsense but as you are aware, the link to d/l the script was broken.
I'll let them know that the link is now operational and to keep an eye out for further instructions and improvements to the scripts.
-Kevin N.
Anybody else use Pocketsense to download from E-Trade? E-Trade started returning null (empty) statements about two weeks ago.
ReplyDeleteFidelity Credit Card: Anyone have a method to connect to this using PocketSense? It is run by Elan Financial. I am told that Quicken can interface, but they are not listed as having an OFX server.
ReplyDeleteRon, the Fidleity Visa does not have an OFX server, and is not automatically downloaded by Pocketsense as the actual Fidlity accounts are.
DeleteThere is an QFX file hat you can download by web browser. There are 3 ways to get Money to accept. One of those ways is to put that file into the PocketSense import folder. One is to rename to *.ofx. The third is to do a registry entry causing Money to recognize qfx files in addition to ofx files.
Thanks, Cal. I was afraid that would be the case.
ReplyDeleteNoticing Fidelity is failing lately. Since OFXHome is showing failure since the 19th, I don't think it's pocketsense, but wanted to add this comment in case others note the issue. I'm going to ignore, and hope it self corrects in a few days given it seems to be a Fidelity change/issue.
ReplyDeletehttp://www.ofxhome.com/index.php/institution/history/449
Just worked for me. This is my entry in sites.dat (gt and lt changed to .):
Delete.site.
SiteName : Fidelity Investments
AcctType : INVSTMT
fiorg : fidelity.com
fid : 7776
url : https://ofx.fidelity.com:443/ftgw/OFX/clients/download
bankid :
brokerid : fidelity.com
appid : QWIN
appver : 2400
mininterval: 90
timeOffset :
./site.
Thanks Cal and Andrew. Apparently it was old connection info as I matched Andrew's and then had no issues.
DeleteHi all,
DeleteI was having issues with Fidelity Net Benefits (they administer my wife's 401(k) plan) starting early last week. I just added the :443 to the URL and it now works properly again.
I can confirm the same fix for Net Benefits as well. Strangely I had :443 on Fidelity but not on Net Benefits. NB stopped working about a week ago. Just added :443 and all is well again.
Deletemabrownmab: Fidelity has been working great for me too. Download 5 times per week with this morning the latest.
ReplyDeleteIf you are having a problem, the contents of the ofx file in the XFR folder may tell you useful troubleshooting info.
Unfortunately OFX Home has shut off getting new stuff, and expects to go off in February unless somebody picks it up. http://www.ofxhome.com/ofxforum/viewtopic.php?id=49843
Thanks Cal. XFR just indicated I wasn't authorized for that server. It had worked for past ~6 months w/o issue and had just stopped in the past couple days, thus I suspected something on Fidelity side.
DeleteMake sure you can still log into the website.
DeleteDoes your OFX file start with these lines?
=======================
OFXHEADER:100
DATA:OFXSGML
VERSION:102
SECURITY:NONE
====================
Cal, I was able to get to a solution. A couple responses above this conversation, Andrew indicated his sites.dat setup. Andrew and you gave me indications that Fidelity was still working, so I modified my sites.dat to match his and that worked w/o issue.
ReplyDeleteI thank you both for the guidance and responses.
https://pocketsense.blogspot.com/2021/08/dev-testing-continued.html?showComment=1637536833702#c1229287313252761250
Problem setting up a Chase credit card.
ReplyDeleteThis is the first time I have tried to use the PocketSense scripts with a Chase credit card, and I am getting this message when I try to use Setup to add the Chase credit card:
An error occurred requesting accounts from the site. Please check username and password.
I have logged into the primary Chase card holder account and enabled desktop software access:
Menu > Security & privacy > Desktop software
I am running PocketSense Version 2021-Aug-18 DEV
Here are my Sites.dat parameters:
SiteName : Chase Visa
AcctType : CCSTMT
fiorg : B1
fid : 10898
url : https://ofx.chase.com
bankid :
brokerid :
ofxVer : 103
appid : QWIN
appver : 2700
mininterval :
timeOffset :
delay :
skipZeroTrans:
userAgent :
dtAcctUp :
clientUID :
Thanks for any help.
Paul White
Hi Paul,
DeleteDid you receive an email from Chase requesting that you confirm your identity?
Check the secure message center on the Chase site for it.
It's simply a matter of clicking a link so as to confirm your ID.
-Kevin N.
Hello Kevin,
DeleteI did not receive an email or a Chase Secure Message. I already had a Chase log in set up for a checking account, and there are some Secure Message for that. The Chase credit card showed up in the Chase checking account log in, and when I tried to register the credit card, I received a message saying it was already registered, so it seems like that part if working as it should. I do not download the Chase checking account transactions, so this is my first attempt to download anything from Chase.
- Paul W.
With very similar settings in my sites.dat, I've had no problems.
ReplyDeletePerhaps your issue is elsewhere?
SiteName : Chase (credit card)
AcctType : CCSTMT #credit card
fiorg : B1
fid : 10898
url : https://ofx.chase.com
bankid :
brokerid :
ofxVer : 103
appid :
appver :
mininterval:
Hi Ron,
DeleteYou don't have an appid or appver, so maybe I should try it with those blanked out.
- Paul
See if my article is helpful: https://microsoftmoneyoffline.wordpress.com/2018/01/07/chase-ofx-downloads/
ReplyDeleteI looked at that before setting up the Chase credit card settings and have looked at it several times to see if there are any clues about what to do. I will try contacting Chase to see if I can get any help on this.
Delete- Paul
I'm trying to create a scrublet for Bank of America, because I want to remove the MEMO line. (replaced < with ( ) but it doesn't work - I don't know Python.
ReplyDeletefrom control2 import *
from scrubber import scrubPrint
import re
def scrub(ofx, siteURL='bankofamerica', accType='CCSTMT'):
Change (MEMO) to (MEMO)
scrubPrint 'cleared MEMO field'
#must return ofx
return ofx
I am going to have to study a bit to do what I intend. I want one that will deal with <MEMO>blah blah blah</MEMO> as well as those not having the end tag.
DeleteBut if you want one that assumes there is always an end tag or never an end tag, that will be faster.
Thank you Cal. Yes I see what you mean - there is no end tag. Let's assume BoA never uses end tag I guess.
Deletehttps://microsoftmoneyoffline.files.wordpress.com/2021/11/memo.png
This comment has been removed by the author.
DeleteThis comment has been removed by the author.
DeleteSorry about the earlier mistakes. I hope it is right this time. I wish I could remove my earlier attempts.
Delete#remove <MEMO> and everything following up to the next <. If that next < starts </MEMO> then remove that too.
(ofx,subcount)=re.subn(r"<MEMO>[^<]*(</MEMO>)?","",ofx)
num_removed=str(subcount)
scrubPrint(num_removed+" MEMOs removed")
Thank you Cal. I hadn't woke up until you did the last post ;)
DeleteIt works great! One thing I've noticed is that all of the 'scrublets' seem to work for everything, rather than just the intended url, but all is well.
Use scrub_discover.py (from the DEV zip file) as an example.
DeleteNote that the scrub() function (started in line 7) only calls the _scrubDiscover() manipulation routine in line 9 if the if 'DISCOVERCARD' in siteURL condition is met. Otherwise it returns ofx unchanged in line 10.
Similarly, for your function to only act on BOA, you would want a similar test. You could test for the URL and accType if you want, so as to treat credit cards diffrently from a BOA bank account.
Thank you once again. That was a good example ;)
DeleteBy the way Cal, it looks like Robert activated the delete comment option for us ;)
DeleteThis comment has been removed by the author.
ReplyDeleteHELP!
ReplyDeleteI installed PocketSense earlier this year after the other guy's approach disappeared and after I got tired of manually entering the previous day's closing stock price. (I'm only looking to update the previous day's closing price and usually update Money in the morning before the market opens.) Initially it did not work properly, but I read somewhere on this blog that I should try changing the defaultInterval in sites.dat to 2. It did work for a while (except on weekends), but recently I had to use a defaultInterval of 1. That works - sometimes. I think last week it worked 4 times. However, this morning (Saturday) I tried and Money does not update. I tried changing the Interval to 2 and it did not work either; I tired 1 again and still no update. Mind you that for each test I would do a Money backup restore of the Friday file - the last time I used Money. I guess that I don't understand what the d'Interval setting does. I have noticed that when it does update correctly, the price shown is from the previous day's close even though the posted date is the date of the update. I guess that's OK as long as it's understood. So, is there some other setting that needs to be set or any ideas as to what's wrong?
I think there's a couple of things going on here:
Delete1) I think that Leo is trying to use PocketSense to update his stock quotes.
2) I think that Leo wants to have his stock quotes updated with the "closing price" at the end of each day.
I think he might be running into MS Money's behavior where only the first quote import of a day is kept. That is, if you get your stock quotes Friday morning, MS Money will put in a quote update for 12/10/2021 with the price at that moment. If you run the scripts again either later Friday or on the weekend where the date/time of the quote still reports Friday, then MS Money will not update the 12/10 quote and it just maintains the previously quoted value.
Workarounds:
1) Only update stock quotes after the close of market.
2) Write your own scrub or quote utility to ensure that the date/time of quotes is input in such a way to maintain the most recent price as the latest quote.
Personally, I don't mind if the quotes are off by at most a day. If this behavior is important to you you'll have to determine what solution will work best for you.
Thanks for the reply.
DeleteYour description of first quote of the day operation makes sense; I knew it worked that way, but I didn't realize that the weekend days were considered to be the same as Friday. Anyway, while researching this problem I found an executable from Gaier Software that did exactly what I wanted. The only bad thing about it, if you want to call it that, is that sometimes some of the quotes from MSN do not match the closing prices from Fidelity and Google. No big deal, if it is off, it's only pennies.
Thanks again.
Leo R, I did not understand much of your post. But I can explain DefaultInterval.
ReplyDeleteWhen PocketSense asks for the data, it asks how many days back to get. If you ask for 30, that's ok. If you ask for 7, that can be ok if you downloaded info within the last 7 days.
Inside each descriptor in sites.dat you can specify minInterval. So if you said minInterval:60, then PS would ask for the last 60 days worth. Try 30 or 60 days.
defaultInterval is the number of days to ask for if the descriptor for the site did not ask for a larger number. I am not sure what happens if the default interval is bigger than min interval. Try 30 or 60 days. I don't see a problem going bigger.
Anyway, 1 does ot seem like a good number to use. I would go for at least 7, and for the occational user, 60 might be bigger. Asking for a number bigger than what the server has won't hurt.
Regarding the stuff I did not understand, I don't know who the other guy was. I don't know what financial institution you are using. And I am really confused as to why you would keep restoring Friday's file. I don't need to know those things, but if you want help with something, you might identify what "not work" means. Do you get an error message in the ofx file on the screen or in the ofx file in the xfr folder? Also, it is usually useful to identify what financial instition you are trying to access. Somebody may have some familiarity.
Also it may be useful to identify what version of PocketSense you are using. If you run setup.py, there will be a header. It might say "PocketSense OFX Download Python Scripts, Ver: 2021-Aug-18 DEV" or something else.
Thanks for your reply.
DeleteSee my reply to Andrew.
The reason why I would do a restore is that I didn't know if Pocket Sense did something "bad" to my file, i.e., I want to start with a known good working version. My version - PocketSense OFX Download Python Scripts, Ver: 28-May-2018. I thought it was the latest. Are you suggesting that there is a newer version?
Leo R: There is what could be thought of as a Release version. https://sites.google.com/site/pocketsense/home/msmoneyfixp1/p2 has that, and I think that is where you are. No need to change unless you want to be leading edge or have the need of a scrubber for a specific FI.
DeleteThere is a also a Development version, like a beta or even pre-beta (but not alpha). http://pocketsense.blogspot.com/2021/02/dev-testing.html describes that. I think the purposes include the ability to make adding FI-specific scrubbers easier.
It is possible to have both on your computer at once.
For the quotes, the Gaier software is very good. It can be run more than once through the day, and the new prices will replace the earlier prices for a day. With PocketSense, the prices come in in the OFX file, and those prices are set. A later download for the same day will not overwrite the first prices. Thus for best pricing records, you should use PocketSense after the market is closed plus whatever time it takes for the quote server to get settled.
Robert,
ReplyDeleteYou paypal coffee fund in the about me link is no longer working...
I really apricate this software.
Thank you for letting me know-- it's really not expected that anyone will buy me coffee for helping here, but that's exactly what I've used it for... coffee! :|
DeleteHas anyone used Etrade Brokerage recently? I've used it for years, but it stopped working in Q4'21 and I'm wondering if anyone knows a fix/workaround. Did they cut off access like Bank of America did a few years ago?
ReplyDeletehttps://www.ofxhome.com/index.php/institution/view/446
Thanks
-Jason
@JasonJ - I ran into this problem last March and again October. According to my previous post from March it looks like the solution is pretty easy:
Delete"The new default user agent is not liked by E-Trade. Setting up a site-specific user agent of "none" or "PocketSense" resolves the issue."
As of today my Etrade brokerage downloads are working fine.
Hi Jason,
ReplyDeleteI don't have an E-Trade account, but I tried setting one up in Quicken. It seemed to work
except that I didn't have a valid account and password. However, by examining the ofx
log I deduced the following site configuration:
site
siteName : E*TRADE Brokerage
acctType : INVSTMT #investment
fiorg : E*TRADE Financial #Try leaving this out if any problems
fid :
url : https://ofx.etrade.com/cgi-ofx/etradeofx
bankid :
brokerid : etrade.com
ofxVer : 102
appid : QWIN
appver : 2700
mininterval :
timeOffset :
delay :
skipZeroTrans:
userAgent :
dtAcctUp : 19900101
clientUID :
site
Hope this works. David
This comment has been removed by the author.
ReplyDeleteThis comment has been removed by the author.
ReplyDeleteThis comment has been removed by the author.
ReplyDeleteOFX scrubbing using ElementTree
ReplyDelete===============================
To Robert and other PocketSense OFX script users,
I hereby post a link to a etree_ofx.py, a module for directly manipulating OFX data within python. It uses ElementTree which is a native python container object, designed to store hierarchical XML-like data structures in memory. Of course, OFX is very much like XML except that it allows constructs that XML forbids. The most notable issue is that OFX data tags can be open. For example the OFX tag containing the number of shares purchased in an investment transaction would be <UNITS>23.858. In XML this would have to be <UNITS>23.858</UNITS>.
Module etree_ofx.py is designed to close the open OFX tags and load the OFX file into an ElementTree which provides tools for directly manipulating the OFX data ( https://docs.python.org/2.7/library/xml.etree.elementtree.html ). This makes it very easy to write sophisticated scrubbers. Fortunately MS Money will directly import OFX files with closed tags.
The link below points to a zipped directory with all the required files to run PocketSense OFX scripts with this functionality added to the the most recent "210910 - beta" release of PocketSense OFX python scripts. You need to add your own sites.dat file and ofx_config.cfg file before running Getdata.py.
This software is version 1.0 beta, intended for experienced users to test and evaluate. Be sure to back up your Money file first. I would suggest using a test version of your Money file until you are comfortable that it works as expected.
I don't recommend it for general use until such time as Robert posts it officially (if he chooses to do so at his discretion). This is a fairly major change to the scrubber process and may involve more work to evaluate than he feels is justified.
For more information on why I created etree_ofx.py, please refer to "Note to Fellow OFX Scrubbers.txt" in the zip file.
https://drive.google.com/file/d/1Atu2hAI5H8mSJDny1ZO_eaG2Mysa4xBU/view?usp=sharing
Happy scrubbing,
David Casada
Updated archive to Version 1.1 using same link as above:
Deletehttps://drive.google.com/file/d/1Atu2hAI5H8mSJDny1ZO_eaG2Mysa4xBU/view?usp=sharing
18Mar2022*dbc Ver 1.1 beta
- Added several new functions to etree_ofx.py
- "Drag-Drop to IndentPrint and Save OFX file as text.py" script creates an
indented copy of any OFX file that you drag and drop onto it. Makes OFX
easier to read.
- Found that not all files were based on the 210910 beta release. Reviewed
and synchronized everthing to use 210910 beta release as the starting
point for adding Ver 1.1 ElementTree functionality.
- Tested etree_ofx scrubbing on OFX files from Vanguard, T. Rowe Price,
Fidelity, Fidelity NetBenefits, and USAA Federal Savings Bank (checking
and credit cards).
Cheers,
David Casada
Posted Version 1.2 beta of the OFX scrubbing utilities using ElementTree as
Delete"PocketSense OFX etree_ofx Ver 1.2.zip" at link:
https://drive.google.com/file/d/17WEZO23n_cBtqPqhtyTMdimGbsF1pWiH/view?usp=sharing
15-May-2022*dbc Version 1.2 beta Changes:
- etree_ofx.py Ver 1.2 beta
Reduced verbosity by Adding "if Debug:" to print statements in InferSite().
- scrubetree_FidelityIncBuyToReinv.py Ver 1.1 beta
Minor update to quietly handle a wider range of simultaneous transaction types.
- scrubetree_FidelityBondAccruedInterest.py Ver 1.0 beta
New scrubber to handle accrued interest for bond purchases, solving two problems:
- Accrued interest with a bond purchase must be a negative number. Fidelity reports it as positive.
- MS Money imports accrued interest before recording the bond purchase. This generates warnings.
As before, this is still beta software even though I have been using it since early March 2022 with investments at Fidelity, Vanguard, and T. Rowe Price. It is intended for experienced users to test and evaluate. Be sure to use a test version of your Money file until you are comfortable that it works as expected.
Cheers, David
David: I didn't study this closer before as I was "quite busy", but thanks much for sharing. I use the etree module in most things xml, but never considered translating the ofx 1.x messages into fully formed xml (ofx 2.x). The scripts do support ofx 2.x xml messages as-is, but as you know, scrub routines were written to work on either and use extensive regex.
DeleteI like the notion working w/ xml, but I'm hesitant to make the change at this point. It's inevitable that something will break that "worked before"... at least that's my experience :|
-Robert
Hi Robert,
DeleteI understand completely your concern with breaking things that worked before and you are responsible for many users who rely on your tools. Indeed, I have not tested etree against any OFX 2.0 messages so there is a risk. However, I continue to find major issues with OFX message formatting at Fidelity, especially related to bond investments where they report transactions that Money ignores due to erroneous configuration. Fidelity also leaves many transactions implied and they have to be created from scratch. I have found that using etree makes it so much easier to fix these problems (I would not be able to do it with regex). Therefore, I will continue to develop a parallel branch current with your latest version for my own use and for anyone else who needs these capabilities and is willing to do their own testing.
Thank you for your continued support of the community.
Cheers, David
Thank you Carlos and David. I was able to get Etrade working again!
ReplyDeleteGlad to help.
DeleteAnyone use Pocketsense with CIT Bank?
ReplyDeleteI tried using my web credentials and the following, with no success:
SiteName : CIT BANK
AcctType : BASTMT
fiorg : 447
url : https://cibng.ibanking-services.com/eAM/Credential/Index?
fid : 61383
bankid : 124084834
brokerid :
appid :
appver :
mininterval :
ofxVer :
skipFailedLogon:
From banking.cit.com: CIT is now a division of First Citizens Bank
ReplyDeleteThis is the MoneyDance entry:
{
"id" = "md:1382"
"uses_fi_tag" = "y"
"access_type" = "OFX"
"fi_name" = "First Citizens"
"bootstrap_url" = "https://www.oasis.cfree.com/fip/genesis/prod/01849.ofx"
"fi_id" = "1849"
"app_id" = "QWIN"
"fi_org" = "First Citizens"
"app_ver" = "2400"
}
You might give that a try after verifying for yourself that I have the correct CIT Bank.
Thanks Andrew. I had already tried that too. One person told me that they had difficulty using Quicken, so I'll just use the qfx export option they offer.
ReplyDeleteHi! IS anybody having trouble with Fidelity (the Netbenefits side: https://nbofx.fidelity.com/netbenefits/ofx/download)? For the last two days, it is coming back to me with a text file that says "THE SERVER ENCOUNTERED AN INTERNAL ERROR OR MISCONFIGURATION AND WAS UNABLE TO
ReplyDeleteCOMPLETE YOUR REQUEST."
Any thoughts?
Harold
I just tested my Fidelity NetBenefits account transaction download, and it was successful. I recall having to add ":443" to the middle of the URL recently. Perhaps that could be the problem for you.
DeleteHere is my sites.dat configuration:
SiteName : Fidelity NetBenefits
AcctType : INVSTMT #investment
fiorg : nbofx.fidelity.com
fid : 8288
url : https://nbofx.fidelity.com:443/netbenefits/ofx/download
bankid :
brokerid : nbofx.fidelity.com
ofxVer :
appid :
appver :
mininterval:
timeOffset :
Cheers, David
Thank you very much. I went to change my Fidelity entry, but first tried it again without changing it. And it worked. Isn't software great? ..... Thanks for your help!
DeleteHarold,
DeleteI had the same issue with the same provider (Fidelity Netbenefits) for roughly the same period of time. I had already put the "443" in the URL from a previous failure. There must have been a site issue for a few days. My account is working again too.
-Andrew
Hello All, I am also an active user of MS Money till date (had been using since last ~20yrs). I was looking for a way to automate the updation of stock/mutual funds quotes in Money investment accounts. I bumped into this very nicely written tool and article. The question I had is, is it necessary to download statements from various providers? Can I use this to just get the quotes from Yahoo/Google and get it updated to Money?
ReplyDeleteThanks in advance.
Pravin
Hi Pravin,
DeletePocketSense scripts can be used to update stock and mutual fund prices without having to download account balances/transactions from each institution. Just set up the tickers you want in the sites.dat file and run Getdata.py. You need to install python 2.7 (not version 3.x) to use these scripts. ActiveState https://www.activestate.com/products/python/ will provide an installation with all needed libraries.
If you just want to update stock and fund prices in Money, there is another excellent tool called MSMoneyQuotes by Gaier Software at https://gaiersoftware.com/Money. It is $10 for a lifetime license and works really well. I use PocketSense python scripts to update transactions and MSMoneyQuotes to update prices.
Good luck,
David
I'm hoping someone can help me out. If this is the wrong place I apologize. I've been using Microsoft Money since it was free and PocketSense has been a lifesaver. I have a report set up for my Apple stock purchases. It wasn't always showing the buy and sell icons but I found a solution somewhere and it fixed the problem. I believe I had to modify each transaction in some way for the icons to show up. Anyway I did a sell a week or so ago and the icon doesn't show up. Anyone have a solution? Thanks!
ReplyDeleteJoe: I never get icons on my reports. I use advanced reports, and have not tried "Essential Reports" for a long time. I wonder if essential reports have icons.
DeleteI tried switching to Essential Reports and my Apple report disappeared so I switched back to Advanced Reports.
DeleteGood morning! Is anybody besides me having trouble with American Express downloads? I get a null (empty) file when I try to download. Harold
ReplyDeleteI have the same issue this evening. When I log on to the AMEX site via the web, it asks for the last 4 of my SSN (every time I log in).
DeleteThe Amex problem with downloads seems to have cured itself. Their online system did ask me for the last four digits of my SS# when I logged in.
DeleteGood morning. After being in a nice steady state for a while my bank was purchased by M&T and their system conversion is now complete. After using Setup to add and test the M&T account, setup says test completed successfully, but when looking at the M&T response in wordpad (in xfr folder) the response looks like some kind of welcome/intro response and definitely not a list of transactions.
ReplyDeleteHas anyone successfully added a M&T bank account?
Hi Michael,
DeleteM&T's system may require reauthorization of your account(s) to allow personal finance management software (pocketsense, quicken, moneydance etc.) access to their servers.
Look to M&T's website to see if there's any means of authorizing your account(s).
You could try doing a Search of their website for 'Quicken' to see if it provides any pointers.
-Kevin N.
Thank you for the reply. I found my conversion notices and there was one regarding Quicken. Seems I have to call to enroll and it seems I get a temporary password that will need to be changed on first use. If I remember correctly from my old bank I had to do that in Moneydance and then grab the connection key from Moneydance and use with pocketsense scripts. Sound right? Is that still the case?
DeleteAfter going through customer service phone hell, I no longer think what I wrote is accurate. Maybe just using another bank will be easier than figuring this out
DeleteHi Michael,
DeleteI'm not an M&T client so I can't speak to what will be required once you've authorized your accounts.
I use Moneydance along with MS Money. Although I CAN use Moneydance directly to download & import, I prefer to use Pocketsense.
I'm not 100% sure what you mean by 'grab the connection key from Moneydance'. You WILL need the particulars for M&T to enter into sites.dat. Perhaps if you can make a successful connection using Moneydance, you can use Stuart B's Toolbox extension to extract the needed sites.dat particulars.
-Kevin N.
Hi, I am now thinking M&T only connects directly to quicken. Moneydance cannot connect and the parameters from ofxhome.com do not work either. The parameters from ofxhome seem to be some kind of ofx test site, maybe for checking if server is up (I suspect anyone can try the ofxhome M&T parameters and just give a made up user/pass, and you will get same response from M&T). Using the ofxhome parameters in sites.dat results in Setup saying successful test, but in reading the response it really looks like a test response only.
DeleteDifferent Question - M&T online site let me download a qif file which I put in /import. But script saw file in /import and ignored this file. When I opened the qif file in notepad there wasn't any account identifying info like bank name or account number, only date, transaction, description. Is that why the pocketsense script skipped because it had no account related info?
I was able to double click the qif file and money then asked me which account and the transactions loaded but I get the sense money will not know next time which account because there is no account identifiers in the file, but at least it is not totally manual
We will see how it goes, might be easier just to use another bank I know currently works.
For the group. One thing I have setup is some batch scripts to deal with missing account number. I save problematic ones as certain file names (like MTBank.qif) and then run a batch file consisting of a powershell text replacement scheme to input the account number.
DeleteSave the below text into a text file and give it a *.bat extension. Put it in the same folder as the MTBank.qif file and run it to do some replacment things. Obvsiously adjust to your needs as sometimes it can be a logic puzzle to try and figure out how to replace the text you want, without messing up the qfx/qif file. Note: I left the first line in to show how you could reqeust info when the batch runs (I had issues with a bank that didn't indicate current balance and I wanted to add it into the qif/qfx file).
The batch just replaces text in the original qif and saves it to a *fixed.qif file. Then if I have more than one text replacements, I just bounce back and forth between *fixed.qif and *fixed-1.qif; over and over. Then at end, del some of the files and rename the final fixed one.
SET /P curbalance=What is the current balance on MTBank?
powershell -Command "(gc MTBank.qif) -replace '0.0', '%curbalance%' | Set-Content MTBankfixed-1.qif"
powershell -Command "(gc MTBankfixed-1.qfx) -replace '', 'Michael'sAccountNumber' | Set-Content MTBankfixed.qif"
del MTBank.qif
del MTBank-1.qif
ren "MTBankfixed.qif" "MTBank.qif"
Well that didnt' work out well. There are missing code from my text area. I'm not sure of the blogger syntax to get this right, but trying one more time.
DeleteSET /P curbalance=What is the current balance on MTBank?
powershell -Command "(gc MTBank.qif) -replace '<BALAMT>0.0</BALAMT>', '<BALAMT>%curbalance%</BALAMT>' | Set-Content MTBankfixed-1.qif"
powershell -Command "(gc MTBankfixed-1.qfx) -replace '<ACCTID></ACCTID>', '<ACCTID>Michael'sAccountNumber</ACCTID>' | Set-Content MTBankfixed.qif"
del MTBank.qif
del MTBank-1.qif
ren "MTBankfixed.qif" "MTBank.qif"
Interesting... the bank provides qif exports but not "Quicken" downloads (i.e., qfx )? Haven't seen that before... they usually offer both.
DeleteGood morning. Yea, download choices are qif and csv
Deletemabrownmab, thanks for the scripts. I tried yesterday and with the revised this morning. The issue is the replace commands - 0.0 is not in the file so command is not working. I think some kind of add command is needed to add %curbalance% but I could not figure out the right syntax for that. Same goes for ACCTID
DeleteSeems like my second syntax appears correct, but I figured you would have to modify the command(s) for what you see in the MTBank qif file. general summary is you are doing -replace THIS, THAT with single quotes around this and that. I'd be comparing MTBank qif file in a text editor to another, working qif/qfx file to see how I could insert the account number using OFX syntax and the text replace commands.
DeleteRobert et al: We will probably hit the 200 post level on this post thread, so I wanted to offer some thoughts on the Dev versions. That sometimes precedes an update. Currently, in the Dev verson, each scrub_*.py scrub file gets processed, and that works well. I had a some alternative ideas.
ReplyDelete1. The sites.dat could add a "scrub_file:" entry type. This could specify the scrub file or files to run for this particular FI. If no such entry, do the same as the current Dev version.
2. If more than one file matches, such as "scrub_file:scrub_fidelity*.py", use the newest maybe? On the other hand, I can see that "scrub_file:scrub_*.py" that matches all could have its merits too. So I don't have a clear preference right now.
3. I could see a case for wanting to pass info when you call... So for example
"scrub_file:scrub_fidelity*.py(1234,'corn flakes')" would pass two parameters for access by argv(). I am a little cautious on this one, because sites.dat does not normally have personal info, and if passing parameters, that could get personal. I like to think of sites.dat descriptors as something that should be shareable.
I know I tend to overcomplicate things at times, but I decided to throw these out for consideration. I already decided that my idea for potential digital signature has too many downsides.
PocketSense has been working great for me as it is now. Janus, Vanguard, Fidelity, Citicard. Banking I only get ofx/qfx via web, which is better than not. If I switch banks, it would be to Chase Bank, which people say works directly with PocketSense. I don't do many banking transactions, so am not that eager to move that. I think it is good to identify particularly well-behaved FIs.
Cal: You reminded me that I still haven't updated the main site w/ the latest versions. That's mostly because the current version has been working well, so my focus has been on other things. Regarding site-specific calls to scrub modules, is there a use case that's not handled by the current method? It's true that all scrub routines get called, but if they don't apply, it's an immediate return w/o action so "no harm".
DeleteThe current Dev system works. I remember it originally confused me, but I now understand it. It gets the job done. My original confusion was probably due to not really reading the notes.
DeleteIn writing a scrubber for the Dev version, I have access to the URL being used and the account type. It has been about a year since I looked at that, but I vaguely remember wishing I had another piece of info.... but maybe I didn't at all.
What's the consensus on continuing to run Python 2.x, given that it's EOL'd and not receiving security updates? I have a new system and I really don't want to install outdated, EOL'd software on it if I can avoid it.
ReplyDeleteI can't speak to consensus, but I'm not opposed to someone making the changes. It won't run as-is under Python 3.x, though, and I don't see myself focusing on it any time soon. I've been working under the assumption that OFX would die before Python 2.x did, but that appears to have been shortsighted ;) I think most of the conversion will be ~ simple, except maybe the way account info is encrypted/stored. That part may require extra effort.
DeletePython.org sunset date was January 1, 2020 Yet the software is still available for download. I see this lack of support to be largely a lack of new features, and the other side of that is stability.
DeleteSo Python 2.7 does not worry me. I use another important piece of software that has been EOLed: Microsoft Money. ;-)
@callearner, the issue is, Python 2.7 has a number of security vulnerabilities that will not be fixed. see https://www.activestate.com/products/python/python-2-end-of-life-security-updates/
DeleteI have primarily Chase accounts, and so have generally not been paying much attention to the new DEV version of Pocketsense. Well, apparently the time has come, since Chase discontinued Direct Download as of today.
ReplyDeleteCan someone do me a big favor and point me to something that will tell me what I need to do in order to get connected back to Chase? I would greatly appreciate it.
Harold
I'm right there with you. It just happened to coincide with a major Windows update, so I was thinking it was Windows that caused the issue. Now I know it's worse. :( Hope I don't have to finally seek out an alternative to Money.
DeleteI was hoping it was just server maintenance, but like Discover Card a few days ago, looks like Quicken has convinced Chase to stop providing direct downloads of your data, unless you are using Quicken software.
DeleteI called the Chase people. As you might imagine, they are having lots of problems with this implementation... apparently whatever they did doesn't easily work with Quicken either. I suggested that if they can't give us back OFX access, they should consider making their regular transaction download-across-accounts allow for OFX format exports (currently it only allows CSV, Excel, or text formats). Of course, the person I was talking to was not empowered to respond to that, but I am hoping against hope that he finds some way to do it with some version of the Chase online software.
DeleteFurther discussion with Chase indicated that they clearly understand the problem now, but they don't have a solution except for account-by-account download using their site. Maybe Robert or someone here will figure something out.
DeleteHarold, the Chase website does offer QFX downloads. See if my blog page helps:
Deletehttps://microsoftmoneyoffline.wordpress.com/2018/01/07/chase-ofx-downloads/
I was all excited to read this, but sadly, I believe that that is exactly the facility that Chase just took away. In their current menu structure, "Profile & Settings > Manage Account Security > Desktop apps > activate Quicken" does not exist... there is no "Manage Account Security". There is something similar, but it doesn't lead to "Desktop Apps". So far, the only way I have found to get OFX downloads from Chase today is to look at a single account, and then there is a Download icon, which offers QFX as a choice. The URL "ofx.chase.com" seems to no longer exist.
DeleteI am also dealing with this change at Chase. My wife and I each have 1 credit card account there. It looks like the OFX server is not online but they do offer "Quicken Web Connect (QFX)" downloads for those accounts.
DeleteMy process for now will be to periodically download those files and put them in the PocketSense "import" folder. I'll need to adjust my scrubbers to make sure they hit these files too.
I have also sent a message to their customer service stating that I have transferred accounts from other institutions when they stopped providing OFX services.
Well folks... It looks like we may be reaching the end of the line here. My Chase downloads are also down (as others have stated, the old server doesn't even show up on DNS any more). More concerningly I found a thread here:
Deletehttps://www.bogleheads.org/forum/viewtopic.php?t=387640
Looks like BoA is also about to kill direct downloads. Mine still work, but the days appeared to be numbered! With no BoA, Chase, or Discover, I'm pretty much out of all my non-investment accounts.
I hate to say it, but slowly Quicken is becoming more and more attractive due to convenience. I'm actually even thinking of writing a letter to Congress and the DoJ. With the elimination of direct downloads PocketSense, MoneyDance and a whole host of 3rd party applications have been monopolied out of existence.
Sad to see this degradation of Chase. I have only used it for the Amazon card, but certainly won't be using it for anything else, after this, unless a fix arrives. I recall USAA had similar issues but Robert and others managed to correct it in time. Hopefully this can be resolved also in the future.
ReplyDeleteI called Chase yesterday to officially complain. I was very nice about it, and the account manager (2nd level support) took the complaint seriously and forwarded it on (after spending significant time making sure she fully understood the nature of the complaint). Will anything happen? Of course not, but I encourage you all to call and lodge an official complaint (nicely)...
DeleteThe PS scripts were built to support online (direct) ofx connections and import to Money, but I'm afraid those days are fading quick. As such, my thoughts lately have been that the "end is nye" for using MS Money w/ direct downloads or ofx import--consistent with what Carlos said above. As a recent example, Discover disabled qfx/ofx downloads by any means at all, meaning that as of ~ last week, Discover account activity cannot be imported to Money. This applies to both the credit and bank accounts.
ReplyDeleteMost institutions support aggregators such as Yodlee and Plaid, plus corporate entities that with which they are willing to establish contractual agreements (e.g., Intuit, Personal Capital, etc.). Of all the options I've seen, though, Quicken is the only one that claims to provide direct import of an existing MS Money data file. I have no experience with it so not sure how well (or if) it works?
Robert,
DeleteIf you try it, please let us know. I see that MoneyDance can also import, but the process seems tedious (export as loose qif; import accounts one at a time; clean things up).
Robert & Money Fans,
DeleteI bought Quicken last year to see if it would connect to USAA when their OFX server (temporarily) stopped working. One benefit of having Quicken is that it maintains a complete log of all its OFX communication. So I have been able to use that log to figure out new OFX server URLs and other configuration data from browsing the Quicken QFX queries and responses. Unfortunately, I don't have Chase so I can't see how they connect to Quicken now. Maybe someone with a Chase account can try the free version long enough to see how it connects. The logs are under the Help -> Contact Support -> Log Files.
As far as importing Money to Quicken, that is another story. I have been a Money user since the very early days and have 30+ years of data. The import process was not clean and I decided it would take far more time than it was worth to fix up the myriad problems reported. That said, I didn't apply myself to solving the problem, and maybe there are just a few transactions that cause the hiccups. I hope to continue using PocketSense and Money until the bitter end. Though Money has long lost its ability to track cost basis for tax purposes, I still know what my Annual % Return performance is and find it useful to track spending by category for budgeting.
As always, a big thanks to Robert (and other active members) for keeping this ship afloat.
Cheers, David
Do you think it's possible to download Chase transactions using a qfx server and have the script rename the qfx file extension to ofx?
ReplyDeleteActually, my app can load in qfx. Just rough that I can't use a script to download the qfx files
DeleteMS Money seems to have no problem with .qfx files, at least the one's generated by my FI's that don't have ofx. I have "associated" .qfx files with MSMONEY so I can just click on them and they will import.
DeleteI regularly download QFX files from my credit union, from Trans America, and from Target Red Card. For TransAmerica and Target I have to run them through a scrubber to fix some issues that makes Money reject them. Otherwise Money has no issues accepting files with a different extension.
DeleteSince Discover Card's boneheaded move to eliminate offering OFX/QFX files for manual download, I'm using the old XL2QIF Excel app to convert Discover Card's CSV file to QIF so as to import it into MS Money. I don't use Discover Card too much so it's not that much of an issue.
ReplyDeleteOn the other hand, Chase Visa, I use on a daily basis for everything. They still offer OFX/QFX files for manual downloads. Their cash-back offers are very generous, so I'll put up with the inconvenience of manual downloads.
I'm not, at all interested in taking the Quicken plunge ever.
@Kevin: Have you used Quicken... or do you just not like the idea of it? Just curious at this point.
DeleteHi Robert,
DeleteI've only used Quicken in the trial-version capacity. My reluctance to wholeheartedly venture into Quicken territory comes from ex-Quicken users who have abandoned it in search of something less $$$ demanding, less buggy and a program that provides some inclining of actual support.
Currently, I'm a longtime Moneydance user (circa 2008) and an even longer MS Money user (thanks to you) who has read the multitude of horror stories on the Moneydance support forums of ex-Quicken users wherein I have no interest in supporting such a company.
-Kevin N.
I don't want to admit it, but I'm at a crossroad with the scripts since we use Discover almost exclusively, and not interested in changing at the moment. The incentive to keep hacking data into Money is more the challenge itself at this point but doesn't make $ense long term, especially when I'm overloaded with challenges elsewhere.
ReplyDeleteBeing a long time Excel user and appreciating having a ledger history, I may check out Tiller Money. It's $80 per year for what amounts to an account aggregation service into Excel or Google Sheets, but the interface seems very simple. I'd guess a decent portion of the cost goes to Yodlee for back-end aggregation, since they rely on them for that part. Historical data is easily exported in csv from Money, so the idea of migrating to worksheets doesn't seem difficult. This option isn't for everyone, but for folks that spend more time in Excel than Word (like me) it seems worth considering.
In general, I suspect that the trend by large banks to make it harder for customers to access their own financial data is emblematic of the way in which big business routinely treats its customers. It amazes me how these decisions seem to be made in both a technological and business vacuum, with no concern evidenced for customer satisfaction. The fact that the banks subsequently try to pretend that they are doing it because of security concerns is beyond laughable. Doesn't allowing their customers to directly see an automated transaction register of their own data ADD to security, as opposed to introducing new third parties who themselves are surely security risks?
DeleteSince October 6th, I have been randomly downloading some of my Chase bank accounts manually and importing the data into Money. Surprisingly, it hasn't really been that bad, especially because I cherry-pick and only download my active accounts . I've been very spoiled by your efforts on Pocketsense over the past years, Robert, and really appreciate them. But I can't see any way we can hope to exert any influence except perhaps via Tick Tock videos .
I personally always hope against hope that there will be some businesses that "get it". And then Google kills off another of its software products, and Discover decides that QFX/OFX is unnecessary, and the vision of a world where cars drive themselves using software created by these types of businesses gets scarier and scarier.
But I did want to take this moment to say "thank you" to you, Robert, for your help, your efforts, your time, and the incredible results over the years since Microsoft abandoned Money.
Thank you Harold. -Robert
DeleteRobert:
Delete1. A CSV should work well for a credit card. When Money imports a QIF, it identifies a duplicate if the payee, amount, and date all match. The flaw is that if there are two transactions where those are the the same, it considers that to be a duplicate, and ignores it when read in as a statement.
I think I could make a scrubber, in Python, that translates a CSV file into a credit card QIF file that could be imported into Money. You could too, obviously, but I have more time.
2. Hung Le (HLE) has written a CSV to OFX tool in Java. See https://groups.google.com/g/microsoft-money/c/Y36nMjpTCOM post dated October 7. Python can run a Java program, tho I have never tried it. I don't know how he generates the FITID. Ideally it would be produced by a hash of certain info, so if that CSV line gets processed later, it will get the same FITID.
Thanks Cal... but you're a little late ;) It floated around in my mind and, then... without planning to... I started writing a csv-to-ofx converter (python) that would work w/ Money. I'd only be happy if the process can be fully scripted w/ reliable results in a few clicks. As of now, it seems to work for both discover credit card and bank savings, but is configurable for any bank (I think).
DeleteIf anyone else is interested I can share once I've used it a few times to verify. It works in both Python 2 and 3 and only has one dependency that may not be in a core (basic) distribution.
I played around with Chase's CSV download. It appears that they cannot download more than one "credit account" at a time... and Chase has a QFX download for individual accounts already built-in. It does let you download more than one "bank account" at a time in CSV format, but despite allowing you to sort-of sort by account on the display, the CSV extract always comes sorted by date, with the accounts interleaved.
DeleteIt seems that if I wanted to automate Chase, it would be interesting to explore browser scripting tools that would let me script the download process account by account. This stuff exists for the purpose of testing scripts, but I know nothing about them, and am not quite ready to trust my financial data to such a tool.
Another method to import Discover CSV files is to use the python module csv2ofx. The created OFX doesn't work (needs MsMoney specific headers), but QIF format does work. csv2ofx does have some heavy dependencies, though `pip install csv2ofx` should handle them (I think). The following mapping file works for me.
Delete```
from operator import itemgetter
from csv2ofx.utils import convert_amount
mapping = {
"has_header": True,
"is_split": False,
"bank": "Discover",
"currency": "USD",
"delimiter": ",",
"account": "MyDiscoverAccount",
"account_id": "HashedDiscoverAccount#",
"date": itemgetter("Post Date"),
"date_fmt": "%m/%d/%Y",
"amount": lambda r: convert_amount(r["Amount"]) * -1,
"payee": itemgetter("Description"),
}
```
Separately, I called Discover to complain about their change, and the support person told me the reason for removing the OFX option is because Quicken told them to.
I had played around w/ the github.com/reubano/csv2ofx library before writing something myself. I found a few issues, beyond the header. Actually, it took a fair bit of editing the ofx output for Money to accept, since it was missing sections and used incompatible formats for some parts. I thought I'd just modify it to fix the issues, but realized it was *very* dependent on other library packages the author had written and made extensive use of itemgetter. It seemed too abstract/complicated and difficult to follow (imho), although it's certainly well written. My personal shot at it is ~ linear and relatively simple, but I've only used it a couple times so far. Need to clean it up a bit...
DeleteI tried downloading the latest package of scripts and was confronted with a 403 not authorized.
ReplyDeleteany assistance would be greatly appreciated.
Thanks,
Are we talking about the "dev" link at the top of this thread, or the "release" version @ https://sites.google.com/site/pocketsense/home/msmoneyfixp1/p2 ? I updated the release version recently so may have broken the link to one (or both?).
DeleteUggggg so upset about the Chase OFX stopage! I used them for two cards for years and it worked great.
ReplyDeleteSo I still use Amex as my primary card and that works fine, but I had a chase amazon card and backup visa with them (as not everyone accepts amex).
What other credit cards still have direct transaction downloads out there so I can switch to them as my backup card?
Citi cards still work.
DeleteThank you! Going to switch to Citi then! I emailed Chase and got some canned response about that my quicken software will automatically update blah blah blah. Useless...
DeleteOk, new Citi card on its way. I was worried about my Chase 5% amazon card, but it turns out Amex has a 5% business card so I just applied for that with my sole proprietor "business" and was approved. So getting a few bonuses, my OFX back, and can actually follow through with my email that i am leaving Chase since they closed for direct connect.
DeleteOh no... BoA connections are currently "active refused" by the target machine. Hoping it is a scheduled maintenance issue, but I fear the worst...
ReplyDeleteI didn't take advantage of using their ofx server while it was available, so I never did BoA with Pocketsense, but I confirmed that the web downloads are still working.
DeleteLooks like they have indeed killed it. Happy Halloween! :(
Deletehttps://infinitekind.tenderapp.com/discussions/online-banking/24496-they-are-dropping-like-flies-direct-connect
Nov 5 and BoA is still down. OFX downloads are dead, Jim. At least Vanguard and Fidelity still work for now... I expect it won't last long.
ReplyDeleteLadies and Gentlement it has been a pleasure downloading with you. Last one out, please turn off the lights.
I sent BoA some "feedback" on this and the executive membership team called me back within a few hours. Essentialy went like this: "We're sorry you're upset about this change. We've noted your complaint, but it won't change a thing". The agent was really nice about it which I appreciated.
DeleteIs anyone looking at Plaid.com.... Their API appears to work with most of the banks in question.... Not nearly as ideal, but if we can create a Json to OFX (or qif) converter it may be a workable solution...?
Before I spend too much time looking at potential solutions like CSV2OFX or Plaid, screen scraping, etc. Have people successfully used Money with Win11? I'm still on Win10, but a new computer is in the future.
ReplyDeleteMoney sunset works perfectly fine on Win11
DeleteNot trying to convince you to stick with Money... but fyi, I'm currently running Money on Win11 Home. In fact, I didn't do anything special but install it... meaning, no dll issues, or any of the hiccups I recall w/ early Win 10.
Delete