Imam Qasim (a.s) and the Wilayah of Imam Ali (a.s)
We addressed part of your question here . In addition to that, I would say that the statement and position of the aforementioned imams is not ijtihadi, rather it is textual. Because the wilaya of Ali (as) is explicitly proven from the Qur'an and mass-transmitted ahadith, to deny it would be akin to denying any other belief supported by the Qur'an and mass-transmitted ahadith.
Allah declares Ali's wilaya in Q. 5:55 as agreed upon by Quranic exegetes. The Prophet, peace and blessings be upon him and his progeny, declared his wilaya in an agreed upon and mass-transmitted hadith of Ghadir. In one variation, the Prophet is narrated to have said: ((He is the wali of every believer after me)). So this would mean that if you are a believer, Ali is your wali. It would also mean that his wilaya is a condition of belief; otherwise the Prophet, peace and blessings be upon him and his progeny, could have said: "...the wali of every Muslim after me." There is also the proof of the consensus of Ahl al-Bayt. The text Anwaar al-Yaqeen states explicitly that it is the consensus of the Ahl al-Bayt that the belief in the imamate of Ali is an individual obligation.
As for the non-acceptance of the acts of worship for those who reject Ali's wilaya, it is narrated in the Amāli of al-Murshid Billah, upon him be peace, with a complete chain of narrators to Ja’far as-Sādiq—his father, Muhammad al-Bāqir—Ali b. al-Hussein—his father, al-Hussein—‘his father, Ali b. Abi Tālib, upon them be peace:
I heard Allah’s Messenger, peace and blessings be upon him and his progeny, say: ((Even if a slave worshipped Allah for seven thousand years during the life of this world but then comes to Allah—the Mighty and Majestic—with hatred for Ali b. Abi Tālib, rejecting his right and violating his Guardianship (wilāya), Allah will make his face wretched and sever his nose)).
It is also narrated in Al-Masābīh of Abul-‘Abbās al-Hasani:
Al-Hussein b. Ali b. Abir-Rabi’ related to us with a complete chain of narrators that Hudhayfa b. al-Yamān said: “I saw Allah’s Messenger, peace and blessings be upon him and his progeny, just as you see me. He took al-Hussein b. Ali b. Abi Tālib, upon them be peace, and said: ((O people, the completion of my proof against the mischievous ones after me is the Guardianship of Ali b. Abi Tālib. Whoever abandons the Guardianship of Ali b. Abi Tālib has left my religion! Therefore, your differences in choice after me won’t be recognised)).
And Allah knows best!